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Preface 


This is the latest draft of notes I have used for the graduate course Communication Network Analy- 
sis, offered by the Department of Electrical and Computer Engineering at the University of Illinois 
at Urbana-Champaign. The notes describe many of the most popular analytical techniques for 
design and analysis of computer communication networks, with an emphasis on performance issues 
such as delay, blocking, and resource allocation. Topics that are not covered in the notes include 
the Internet protocols (at least not explicitly), simulation techniques and simulation packages, and 
some of the mathematical proofs. These are covered in other books and courses. 

The topics of these notes form a basis for understanding the literature on performance issues 
in networks, including the Internet. Specific topics include 


e The basic and intermediate theory of queueing systems, along with stability criteria based on 
drift analysis and fluid models 


e The notion of effective bandwidth, in which a constant bit rate equivalent is given for a bursty 
data stream in a given context 


e An introduction to the calculus of deterministic constraints on traffic flows 


e The use of penalty and barrier functions in optimization, and the natural extension to the use 
of utility functions and prices in the formulation of dynamic routing and congestion control 
problems 


e Some topics related to performance analysis in wireless networks, including coverage of basic 
multiple access techniques, and transmission scheduling 


e The basics of dynamic programming, introduced in the context of a simple queueing control 
problem 


e The analysis of blocking and the reduced load fixed point approximation for circuit switched 
networks. 


Students are assumed to have already had a course on computer communication networks, al- 
though the material in such a course is more to provide motivation for the material in these notes, 
than to provide understanding of the mathematics. In addition, since probability is used exten- 
sively, students in the class are assumed to have previously had two courses in probability. Some 
prior exposure to the theory of Lagrange multipliers for constrained optimization and nonlinear 
optimization algorithms is desirable, but not necessary. 

Pm grateful to students and colleagues for suggestions and corrections, and am always eager 
for more. Bruce Hajek, December 2006 


Chapter 1 


Countable State Markov Processes 


1.1 Example of a Markov model 


Consider a two-stage pipeline as pictured in Figure 1.1. Some assumptions about it will be made 
in order to model it as a simple discrete time Markov process, without any pretension of modeling 
a particular real life system. Each stage has a single buffer. Normalize time so that in one unit 
of time a packet can make a single transition. Call the time interval between k and k + 1 the kth 
“time slot,” and assume that the pipeline evolves in the following way during a given slot. 


If at the beginning of the slot, there are no packets in stage one, then a new packet arrives to stage 
one with probability a, independently of the past history of the pipeline and of the outcome 
at state two. 


If at the beginning of the slot, there is a packet in stage one and no packet in stage two, then the 
packet is transfered to stage two with probability d;. 


If at the beginning of the slot, there is a packet in stage two, then the packet departs from the 
stage and leaves the system with probability d2, independently of the state or outcome of 
stage one. 


These assumptions lead us to model the pipeline as a discrete-time Markov process with the 
state space S = (00,01,10,11), transition probability diagram shown in Figure 1.2 (using the 
notation Z = 1 — x) and one-step transition probability matrix P given by 


a 0 a 0 
ads ada ad) adə 
0 d dy 0 


0 O d dy 


d d 


Figure 1.1: A two-stage pipeline 


Figure 1.2: One-step transition probability diagram for example. 


The rows of P are probability vectors. (In these notes, probability vectors are always taken to 
be row vectors, and more often than not, they are referred to as probability distributions.). For 
example, the first row is the probability distribution of the state at the end of a slot, given that 
the state is 00 at the beginning of a slot. Now that the model is specified, let us determine the 
throughput rate of the pipeline. 

The equilibrium probability distribution 7 = (709,701, T10, 7711) is the probability vector satis- 
fying the linear equation 7 = TP. Once m is found, the throughput rate y can be computed as 
follows. It is defined to be the rate (averaged over a long time) that packets transit the pipeline. 
Since at most two packets can be in the pipeline at a time, the following three quantities are all 
clearly the same, and can be taken to be the throughput rate. 


The rate of arrivals to stage one 
The rate of departures from stage one (or rate of arrivals to stage two) 
The rate of departures from stage two 


Focus on the first of these three quantities. Equating long term averages with statistical averages 
yields 


n = Plan arrival at stage 1] 
= Plan arrival at stage 1|stage 1 empty at slot beginning] Plstage 1 empty at slot beginning] 


= a(too + T01). 


Similarly, by focusing on departures from stage 1, obtain 7 = d,719. Finally, by focusing on 
departures from stage 2, obtain y = d2(701 +711). These three expressions for y must agree. 

Consider the numerical example a = dı = da = 0.5. The equation m = TP yields that m is 
proportional to the vector (1,2,3,1). Applying the fact that m is a probability distribution yields 
that m = (1/7,2/7,3/7,1/7). Therefore y = 3/14 = 0.214.... 

By way of comparison, consider another system with only a single stage, containing a single 
buffer. In each slot, if the buffer is empty at the beginning of a slot an arrival occurs with probability 
a, and if the buffer has a packet at the beginning of a slot it departs with probability d. Simultaneous 
arrival and departure is not allowed. Then S = {0,1}, 7 = (d/(a+d),a/(a+d)) and the throughput 
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rate is ad/(a+d). The two-stage pipeline with dz = 1 is essentially the same as the one-stage system. 
In case a = d = 0.5, the throughput rate of the single stage system is 0.25, which as expected is 
somewhat greater than that of the two-stage pipeline. 


1.2 Definition, Notation and Properties 


Having given an example of a discrete state Markov process, we now digress and give the formal 
definitions and some of the properties of Markov processes. Let T be a subset of the real numbers 
R and let S be a finite or countably infinite set. A collection of S-valued random variables (X(t) : 
t € T) is a discrete-state Markov process with state space S if 


P[X (tn+1) = inti|X (tn) = in, ,X (tı) = i1] = P[X (tn+1) = in+11X (tn) = in] (1.1) 
whenever 
ty <tg< ... < tpy1 are in T, 
li, i2, sestri are in S, and (1.2) 
PIXE tein cee Sa SO 


Set pij(s,t) = P[X(t) = j|X(s) = i] and m;(t) = P[X(t) = i]. The probability distribution 
m(t) = (m;(t) : i € S) should be thought of as a row vector, and can be written as one once S 
is ordered. Similarly, H(s,t) defined by H(s,t) = (pij(s,t) : i,j € S) should be thought of as a 
matrix. Let e denote the column vector with all ones, indexed by S. Since m(t) and the rows of 
H(s,t) are probability vectors for s,t € T and s < t, it follows that r(t)e = 1 and, H(s,t)e = e. 

Next observe that the marginal distributions m(t) and the transition probabilities p;;(s,t) de- 
termine all the finite dimensional distributions of the Markov process. Indeed, given 


ty<tg< ... <tp inT, 
{ li, 19, ..., lp € S (1.3) 
one writes 
P[X(t1) Fa 11, , X (tn) = in] = 
PIX (ti) = treg X tn- = tna PX (ta = in| X (N = ii X (taai) = tni] 
= P[X(t1)=41,..., X(tn-1) = in—11Pi,—1in (En—1) tn) 
Application of this operation n — 2 more times yields that 
P[X(t,) = i1, X (to) = 12, sven ,X (tn) — in] = Ti (t1)Piia (ta; t2) mE Die bd) (1.4) 


which shows that the finite dimensional distributions of X are indeed determined by (r(t)) and 
(pij(s,t)). From this and the definition of conditional probabilities it follows by straight substitution 
that 


PIXG,) = ij, for 1 < j < nt) = in] = (1.5) 
PIX (t;) = ij, for 1 < j < n|X (tn) = in] P[X (t;) = ij, for n < j SX (yy) = in] 


whenever P|X (tn) = in] > 0. Property (1.5) is equivalent to the Markov property. Note in addition 
that it has no preferred direction of time, simply stating that the past and future are conditionally 
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independent given the present. It follows that if X is a Markov process, the time reversal of X 
defined by X(t) = X(—t) is also a Markov process. 

A Markov process is time homogeneous if pj;(s,t) depends on s and t only through t — s. In 
that case we write p;;(t — s) instead of p;;(s,t), and Hj;(t — s) instead of H;;(s, t). 

Recall that a random process is stationary if its finite dimensional distributions are invariant 
with respect to translation in time. Referring to (1.4), we see that a time-homogeneous Markov 
process is stationary if and only if its one dimensional distributions m(t) do not depend on t. If, in 
our example of a two-stage pipeline, it is assumed that the pipeline is empty at time zero and that 
a # 0, then the process is not stationary (since 7(0) = (1,0,0,0) % (1) = (1 — a,0,a,0)), even 
though it is time homogeneous. On the other hand, a Markov random process that is stationary is 
time homogeneous. 

Computing the distribution of X(t) by conditioning on the value of X(s) yields that 7;(t) = 
do; P[X(s) = i, X(t) = j] = DO; mils)pij(s,t), which in matrix form yields that m(t) = r(s)H(s, t) 
for s,t € T,s < t. Similarly, given s < T < t, computing the conditional distribution of X(t) given 
X(s) by conditioning on the value of X(7) yields 


H(s,t) = H(s,7)H(r,t) s T,tET, s<7T<t. (1.6) 


The relations (1.6) are known as the Chapman-Kolmogorov equations. 

If the Markov process is time-homogeneous, then r(s + 7) = m(s)H(r) for s,s +7 € T and 
T > 0. A probability distribution 7 is called an equilibrium (or invariant) distribution if 1H (7) = 7 
for all 7 > 0. 

Repeated application of the Chapman-Kolmogorov equations yields that p;¿(s,t) can be ex- 
pressed in terms of transition probabilities for s and t close together. For example, consider 
Markov processes with index set the integers. Then H(n,k +1) = H(n,k)P(k) for n < k, where 
P(k) = H(k,k + 1) is the one-step transition probability matrix. Fixing n and using forward re- 
cursion starting with H(n,n) = I, H(n,n + 1) = P(n), H(n,n + 2) = P(n)P(n + 1), and so forth 
yields 

H(n,l) = P(n)P(n+1)---P(l- 1) 


In particular, if the process is time-homogeneous then H (k) = P* for all k for some matrix P, and 
n(l) = P!Er(k) for l > k. In this case a probability distribution 7 is an equilibrium distribution 
if and only if TP = 7. 

In the next section, processes indexed by the real line are considered. Such a process can be 
described in terms of p(s, t) with t— s arbitrarily small. By saving only a linearization, the concept 
of generator matrix arises naturally. 


1.3 Pure-Jump, Time-Homogeneous Markov Processes 


Let S be a finite or countably infinite set, and let A ¢ S. A pure-jump function is a function 
x: R} —> SU{A} such that there is a sequence of times, 0 = To < T1 < ..., and a sequence of 
states, So, S1, ... with s; € S, and si Æ Si+1, i > 0, so that 


ET AA ; KS 
v(t) = { ae ea oe (1.7) 


A ift>7* 


where 7* = lim;_...7;. If 7* is finite it is said to be the explosion time of the function æ. The 
example corresponding to S = {0,1,...}, 7 = i/(i + 1) and s; = 7 is pictured in Fig. 1.3. Note 
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Figure 1.4: Transition rate diagram for a continuous time Markov process 


that 7* = 1 for this example. A pure-jump Markov process (X+ : t > 0) is a Markov process such 
that, with probability one, its sample paths are pure-jump functions. 
Let Q = (qij : i, j € S) be such that 


qij 20 1jES, i$j 
(1.8) 
Gi = — jes ji Gj ES. 

An example for state space S = {1, 2,3} is 


—1 0.5 0.5 


which can be represented by the transition rate diagram shown in Figure 1.4. A pure-jump, time- 
homogeneous Markov process X has generator matrix Q if 


lm (palh) — Tua (b= ay a5 €S (1.9) 
or equivalently 
pij(h) = Iti=5) + hqij + o(h) 1,] € S (1.10) 


where o(h) represents a quantity such that limy,_0 o(h)/h = 0. 


Y 


For the example this means that the transition probability matrix for a time interval of duration 
h is given by 


1=h 05h 0.5h o(h) o(h) olh) 
h 1-2h h + | o(h) o(h) o(h) 
0 h 1=h o(h) o(h) o(h) 


The first term is a stochastic matrix, owing to the assumptions on the generator matrix Q. 


Proposition 1.3.1 Given a matrix Q satisfying (1.8), and a probability distribution n(0) = (7;(0) : 
i E€ S), there is a pure-jump, time-homogeneous Markov process with generator matriz Q and initial 
distribution (0). The finite-dimensional distributions of the process are uniquely determined by 


Tr(0) and Q. 


The proposition can be proved by appealing to the space-time properties in the next section. In 
some cases it can also be proved by considering the forward-differential evolution equations for z(t), 
which are derived next. Fix t > 0 and let h be a small positive number. The Chapman-Kolmogorov 
equations imply that 


pe ale = Y r(t) (At (1.11) 


h 
1€S 


Consider letting h tend to zero. Jf the limit in (1.9) is uniform in i for j fixed, then the limit and 
summation on the right side of (1.11) can be interchanged to yield the forward-differential evolution 


equation: 
On; (t 
D = lta (1.12) 
iES 
or ao = T(t)Q. This equation, known as the Kolmogorov forward equation, can be rewritten as 
On; (t 
DL E now- ra (1.13) 
¡eSiAj ¡ES iAj 


which states that the rate change of the probability of being at state j is the rate of “probability 
flow” into state j minus the rate of probability flow out of state j. 
1.4 Space-Time Structure 


Let (Xy : k € Z4) be a time-homogeneous Markov process with one-step transition probability 
matrix P. Let Tẹ denote the time that elapses between the kt” and k + 1% jumps of X, and let 
X/(k) denote the state after k jumps. See Fig. 1.5 for illustration. More precisely, the holding 
times are defined by 


T = minft>0: X(0 2 X(0)} (1.14) 
Te = min{t >0: X(To + eh + Tk—1 +!) A X(To + TES + Tr-1)) (1.15) 


and the jump process XJ = (X7 (k) : k > 0) is defined by 


X7(0)=X(0) and X7(k)=X(To+ ... +T) (1.16) 
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J 
x (1 
<a J). yd s 
A O 011111 X ( ) 
1 E OS TEE o m1 k 


Figure 1.5: Illustration of jump process and holding times. 


Clearly the holding times and jump process contain all the information needed to construct X, and 
vice versa. Thus, the following description of the joint distribution of the holding times and the 
jump process characterizes the distribution of X. 


Proposition 1.4.1 Let X = (X(k):k € Z4) be a time-homogeneous Markov process with one-step 
transition probability matrix P. 


(a) The jump process X7 is itself a time-homogeneous Markov process, and its one-step transition 
probabilities are given by Di; = pij/(1 — pa) for i £ j, and p} =0, i,j E€ s. 


(b) Given X(0), XT (1) is conditionally independent of To. 


(c) Given (X7(0),..., Xx (n)) = (jo, -.., jn), the variables To, . . . ,Tn are conditionally indepen- 
dent, and the conditional distribution of T, is geometric with parameter Pij: 


PLD; = k|X7(0) = jo, ... R a ja] =p (1 — pay.) DSS 


Proof. Observe that if X(0) = i, then 
{Ty = k, X7(1) = j} = {X(1) =i, X(2) =i,..., X(k-— 1) =i, X(k) = j}, 


SO 
PM = k, UD) = 4/0) = į] = pf piy =[(1= papi ph (1.17) 


Because for i fixed the last expression in (1.17) displays the product of two probability distributions, 
conclude that given X(0) = i, 


To has distribution ((1 — pudor: : k > 1), the geometric distribution of mean 1/(1 — pi) 
X/(1) has distribution (pj, : j E€ S) (i fixed) 
To and X7 (1) are independent 

More generally, check that 


PIX? (1) = ji, ..., XT (n) = jn, To = ko, . . Ta = ka| X7 (0) = i] = 
n 
Jad J kı—1 
E C) 
l=0 


Figure 1.6: Illustration of sampling of a pure-jump function 


This establishes the proposition. a 


Next we consider the space-time structure of time-homogeneous continuous-time pure-jump 
Markov processes. Essentially the only difference between the discrete- and continuous-time Markov 
processes is that the holding times for the continuous-time processes are exponentially distributed 
rather than geometrically distributed. Indeed, define the holding times Ty,k > 0 and the jump 
process X7 using (1.14)-(1.16) as before. 


Proposition 1.4.2 Let X = (X(t) : t € Ry) be a time-homogeneous, pure-jump Markov process 
with generator matrix Q. Then 


(a) The jump process X! is a discrete-time, time-homogeneous Markov process, and its one-step 
transition probabilities are given by 


J J -%;/% fori Fj 
a { 0 fori=j Ete) 
(b) Given X(0), X7(1) is conditionally independent of To. 


(c) Given X7(0) = jo, . . . ,X7(n) = jn, the variables To, ... ,Tn are conditionally independent, 
and the conditional distribution of T, is exponential with parameter —qj j: 


PIN > e|X7(0) = jo, ..- ,X (nm) = jn] = ezpleg) OSI <n. 


Proof. Fix h > 0 and define the “sampled” process X(W by X(W(k) = X(hk) for k > 0. See 
Fig. 1.6. Then X(”) is a discrete time Markov process with one-step transition probabilities p;¿(h) 


(the transition probabilities for the original process for an interval of length h). Let (Eo :k>0) 
denote the sequence of holding times and (X%*(%) : k > 0) the jump process for the process X”). 

The assumption that with probability one the sample paths of X are pure-jump functions, 
implies that with probability one: 


lim (X*"(0), RIE), Rens AAA), ar, nT”, aoe nT”) E 


CAMA A ie Tn) (1.19) 
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Figure 1.7: Transition rate diagram of a birth-death process 


Since convergence with probability one implies convergence in distribution, the goal of identifying 
the distribution of the random vector on the righthand side of (1.19) can be accomplished by 
identifying the limit of the distribution of the vector on the left. 

First, the limiting distribution of the process X 7? is identified. Since X( has one-step transi- 
tion probabilities p;¿(h), the formula for the jump process probabilities for discrete-time processes 


(see Proposition 1.4.1, part a) yields that the one step transition probabilities ye for X») are 
given by 
7 1 — pu(h) 
pij(h)/h Aj 
> as h— 0 1.20 
(1 — pu(h))/h — Vi vay 


for i Æ j, where the limit indicated in (1.20) follows from the definition (1.9) of the generator matrix 
Q. Thus, the limiting distribution of X%” is that of a Markov process with one-step transition 
probabilities given by (1.18), establishing part (a) of the proposition. The conditional independence 
properties stated in (b) and (c) of the proposition follow in the limit from the corresponding 
properties for the jump process X”? guaranteed by Proposition 1.4.1. Finally, since log(1 + 0) = 
0 + 0(0) by Taylor's formula, we have for all c > 0 that 


PITO > XAO) = jos., X7 = jn] = (Pa A 

c/h] log(p;,;(h))) 
c/h) (dnih + o(h))) 
=> explq;;jc) as h — o0 


which establishes the remaining part of (c), and the proposition is proved. a 


Birth-Death Processes A useful class of countable state Markov processes is the set of 
birth-death processes. A (continuous time) birth-death process with parameters (Ap, A2,...) and 
(pa, ua, ...) (also set A-1 = uo = 0) is a pure-jump Markov process with state space S = Z4, and 
generator matrix Q defined by qkk+1 = Àk, qkk = — (Hk + Àk), and qux-1 = Hk for k > 0, and qij = 0 
if |i — j| > 2. The transition rate diagram is shown in Fig. (1.7). The space-time structure of such 
a process is as follows. Given the process is in state k at time t, the next state visited is k +1 with 
probability A,/(Ax + uk) and k— 1 with probability uz/(Az + uk). The holding time of state k is 
exponential with parameter Az + Hk- 
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The space-time structure just described can be used to show that the limit in (1.9) is uniform 
in ¿ for j fixed, so that the Kolmogorov forward equations are satisfied. These equations are: 


On, (t) 
Ot 


= Ap 11 (8) — (AF pu) mg (t) + Mega TT 41 (t) (1.21) 


1.5 Poisson Processes 


A Poisson process with rate A is a birth-death process N = (N(t) : t > 0) with initial distribution 
P|N(0) = 0] = 1, birth rates Az = A for all k and death rates uz = 0 for all k. The space-time 
structure is particularly simple. The jump process N7 is deterministic and is given by N/(k) = k. 
Therefore the holding times are not only conditionally independent given N7, they are independent 
and each is exponentially distributed with parameter A. 

Let us calculate 7;(t) = P[N(t) = j]. The Kolmogorov forward equation for k = 0 is O1p/0t = 
—Aro, from which we deduce that mo(t) = exp(—At). Next the equation for mı is O7,/0t = 
Aexp(—At) — A(t) which can be solved to yield 7,(t) = (At) exp(—At). Continuing by induction on 
k, verify that m(t) = (At)! exp(—At)/k!, so that N(t) is a Poisson random variable with parameter 
k. 

It is instructive to solve the Kolmogorov equations by another method, namely the z-transform 
method, since it works for some more complex Markov processes as well. For convenience, set 
m_1(t) = 0 for all t. Then the Kolmogorov equations for 7 become ork = ATk—1 — At. Multiplying 
each side of this equation by z*, summing over k, and interchanging the order of summation 
and differentiation, yields that the z transform P*(z,t) of m(t) satisfies OP tt) = (Az — A)P*(z,t). 
Solving this with the initial condition P*(z,0) = 1 yields that P*(z,t) = exp((Az—A)t). Expanding 
exp(Azt) into powers of z identifies (At)F exp(—At)/k! as the coefficient of 2” in P*(z,t). 

In general, for i fixed, p;;(t) is determined by the same equations but with the initial distribution 
pij(0) = Iga}. The resulting solution is p;;(t) = 7;-;(t) for j > i and p;(t) = 0 otherwise. Thus 
for to < tı <...<tg=s<t, 


PIN (t) — N (s) = I[N (to) = to, -.-,N(ta-1) = ta-1, N(s) = i] 
= PIN(t) STINS =i 
= [At — s)]' exp(—A(t — s))/l! 


Conclude that N(t) — N(s) is a Poisson random variable with mean A(t — s). Furthermore, N (t) — 
N (s) is independent of (N(u) : u < s), which implies that the increments N (t1) — N (to), N (t2) — 
N(t1), N(t3) — N(t2),... are mutually independent whenever 0 < ty <ti <.... 

Turning to another characterization, fix T > 0 and A > 0, let U1, U2,... be uniformly distributed 
on the interval [0,7], and let K be a Poisson random variable with mean AT. Finally, define the 


random process (N(t):0<t<T) by 
- K 
N(t) = So lugs: (1.22) 
i=1 


That is, for 0 < t < T, N (t) is the number of the first K uniform random variables located in 
[0, t]. We claim that N has the same distribution as a Poisson random process N with parameter A, 
restricted to the interval [0, T]. To verify this claim, it suffices to check that the increments of the two 
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processes have the same joint distribution. To that end, fix 0 = tọ < ty... < tq = T and nonnegative 
integers k1,...,kg. Define the event A = (N(t1) — N(to) = k1,..., N(ta) — N(ta-1) = ka}, and 
define the event A in the same way, but with N replaced by Ñ. It is to be shown that P[A] = P[A]. 

The independence of the increments of N and the Poisson distribution of each increment of N 


implies that 
d 


P[A] = [At = Eye exp(—A(t; = ti_1)) /kj!. (1:23) 

i=l 
On the other hand, the event A is equivalent to the condition that K = k, where k = kı +... + ka, 
and that furthermore the k random variables U,,...,U, are distributed appropriately across the d 
subintervals. Let p; = (ti — ti-1)/T denote the relative length of the ith subinterval [t;—1, ti]. The 
probability that the first kı points U,,...,U,, fall in the first interval [to, t1], the next ka points 


fall in the next interval, and so on, is pa pe aa pra, However, there are ( ka r ha) = A equally 
likely ways to assign the variables U1,...,Up to the intervals with k; variables in the ith interval 
for all 2, so 
a k k k 
P[A] = P|K = k] Pi -Pa > (1.24) 
ki... ka 


Some rearrangement yields that P[A] = P[A], as was to be shown. Therefore, N and N restricted 
to the time interval [0, T] have the same distribution. 

The above discussion provides some useful characterizations of a Poisson process, which are 
summarized in the following proposition. Suppose that X = (X(t) : t > 0) has pure-jump sample 
paths with probability one, suppose P[X(0) = 0] = 1, and suppose A > 0. 


Proposition 1.5.1 The following four are equivalent: 
(i) X is a Poisson process with rate X. 


(ii) The jumps of X are all size one, and the holding times are independent, exponentially dis- 
tributed with parameter X. 


(itt) Whenever 0 < to < ti... < tn, the increments X(t1) — X(to), ...,X(tn) — X(tn-1) 
are independent random variables, and whenever 0 < s < t, the increment X(t) — X(s) is a 
Poisson random variable with mean X(t — s). 


(iv) Whenever 0 < s < t, the increment X(t) — X(s) is a Poisson random variable with mean 
A(t — s), and given X(t) — X(s) = n, the jumps of X in [s,t] are height one and the vector 
of their locations is uniformly distributed over the set {T © RE :s <<... <mn <t} 


The fourth characterization of 1.5.1 can be readily extended to define Poisson point processes 
on any set with a suitable positive measure giving the mean number of points within subsets. For 
example, a Poisson point process on RY with intensity A is specified by the following two conditions. 
First, for any subset A of R? with finite volume, the distribution of the number of points in the 
set is Poisson with mean A- Volume(A). Second, given that the number of points in such a set 
A is n, the locations of the points within A are distributed according to n mutually independent 
random variables, each uniformly distributed over A. As a sample calculation, take d = 2 and 
consider the distance R of the closest point of the Poisson point process to the origin. Since the 
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number of points falling within a circle of radius r has the Poisson distribution with mean mr, the 
distribution function of R is given by 


P[R < r] = 1-P[no points of the process in circle of radius r] = 1 — exp(Amr?). 


1.6 Renewal Theory 


Renewal theory describes convergence to statistical equilibrium for systems involving sequences of 
intervals of random duration. It also helps describe what is meant by the state of a random system 
at a typical time, or as viewed by a typical customer. We begin by examining Poisson processes 
and an illustration of the dependence of an average on the choice of sampling distribution. 

Consider a two-sided Poisson process (N(t) : t € R) with parameter A, which by definition 
means that N(0) = 0, N is right-continuous, the increment N(t + s) — N(t) is a Poisson random 
variable with mean A for any s > 0 and t, and the increments of N are independent. Fix a time t. 
With probability one, t is strictly in between two jump times of N. The time between t and the first 
jump after time t is exponentially distributed, with mean 1/14. Similarly, the time between the last 
jump before time t and t is also exponentially distributed with mean 1/A. (In fact, these two times 
are independent.) Thus, the mean of the length of the interval between the jumps surrounding t 
is 2/4. In contrast, the Poisson process is built up from exponential random variables with mean 
1/A. That the two means are distinct is known as the “Poisson paradox.” 

Perhaps an analogy can help explain the Poisson paradox. Observe that the average population 
of a country on earth is about 6 x 10%/200 = 30 million people. On the other hand, if a person 
is chosen at random from among all people on earth, the mean number of people in the same 
country as that person is greater than 240 million people. Indeed, the population of China is about 
1.2 x 10°, so that about one of five people live in China. Thus, the mean number of people in the 
same country as a randomly selected person is at least 1.2 x 10°/5=240 million people. The second 
average is much larger than the first, because sampling a country by choosing a person at random 
skews the distribution of size towards larger countries in direct proportion to the populations of the 
countries. In the same way, the sampled lifetime of a Poisson process is greater than the typical 
lifetimes used to generate the process. 

A generalization of the Poisson process is a counting process in which the times between arrivals 
are independent and identically distributed, but not necessarily exponentially distributed. For 
slightly more generality, the time until the first arrival is allowed to have a distribution different 
from the lengths of times between arrivals, as follows. 

Suppose F and G are distribution functions of nonnegative random variables, and to avoid 


trivialities, suppose that F(0) < 1. Let Ti, To, . . . be independent, such that Tı has distribution 
function G and Tẹ has distribution function F for k > 2. The time of the nt” arrival is T(n) = 
T+ 72+ ... +T, and the number of arrivals up to time t is a(t) = maxín : T(n) < tj. The 


residual lifetime at a fixed time t is defined to be y, = T(a(t) + 1) —t and the sampled lifetime at a 
fixed time t is Li = Ty(z)41, as pictured in Figure 1.8. 1 Renewal theory, the focus of this section, 
describes the limits of the residual lifetime and sampled lifetime distributions for such processes. 
This section concludes with a presentation of the renewal equation, which plays a key role in 
renewal theory. To write the equation in general form, the definition of convolution of two measures 
on the real line, given next, is needed. Let Fi and F3 be two distribution functions, corresponding 


¡Note that if one or more arrivals occur at a time t, then y and L; are both equal to the next nonzero lifetime. 
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Figure 1.8: Sampled lifetime and forward lifetime 


to two positive measures on the real line. The convolution of the two measures is again a measure, 
such that its distribution function is Fı x F2, defined by the Riemann-Stieltjes integral 


OO 
F, + Fo(x) = | Fı(x — y)Fa(dy) 
—00 

If Xı and X are independent random variables, if Xı has distribution function F1, and if Xə has 
distribution function F», then Xı + Xə has distribution function Fy x Fy. It is clear from this that 
convolution is commutative (F * G = Gx F) and associative ((Fx*G)x*H =Fx(GxH)). I Fy 
has density function fı and Fə has density function F», then Fy * Fy has density function fi * fa, 
defined as the usual convolution of functions on R: 


fi fala) = f “ERA 


Similarly, if F1 has a discrete mass function fı on Z and F> has discrete mass function fə on Z, 
then Fi x Fə has discrete mass function fı * f2 defined as the usual convolution of functions on Z: 


fix foln) = Y fi(n—k) folk). 


k=—0o 


Define the cumulative mean renewal function H by H(t) = Ela(t)]. Function H can be viewed 
as the cumulative distribution function for the renewal measure, which is the measure on R+ such 
that the measure of an interval is the mean number of arrivals in the interval. In particular, for 
0 < s <t, H(t) — H(s) is the mean number of arrivals in the interval (s,t]. The function H 
determines much about the renewal variables-in particular the joint distribution of (Lt, y+) can be 
expressed in terms of H and the CDFs F and G. For example, if t > u > 0, then 


t 
Pla] f Pla renewal occurs in [s,s + ds] with lifetime € [t — s, ul] 
t 


—u 


i (F(u) — F(t — s))dH(s). (1.25) 


—u 


Observe that the function H can be expressed as 
AO) = Uni) (1.26) 
n=1 


where U,, is the probability distribution function of Tı +- --+Tn. From (1.26) and the facts U; = G 
and Un = F x U,,-1 for n > 2, the renewal equation follows: 


H=G+Fx*H 
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1.6.1 Renewal Theory in Continuous Time 


The distribution function F is said to be lattice type if, for some a > 0, the distribution is con- 
centrated on the set of integer multiples of a. Otherwise F is said to be nonlattice type. In this 
subsection F is assumed to be nonlattice type, while the next subsection touches on discrete-time 
renewal theory, addressing the situations when F is lattice type. If F and G have densities, then H 
has a density and the renewal equation can be written as h = g+ fxh. Many convergence questions 
about renewal processes, and also about Markov processes, can be settled by the following theorem 
(for a proof, see [1, 18]). 


Proposition 1.6.1 (Blackwell’s Renewal Theorem in continuous time) Suppose F is a nonlattice 
distribution. Then for fixed h > 0, 


Jim (A(t +h) — H(t))/h =1/m,, 
00: 
where my is the kt” moment of F. 
The renewal theorem can be used to show that if the distribution F is nonlattice, then 
Jim P[L; < u, y < s] = Fry(u, s) 
—00 


where 


1 s^u 

Fous) = | EO- FO) us ER (1.27) 
0 

For example, to identify lim; P[L; < u] one can start with (1.25). Roughly speaking, the renewal 

theorem tells us that in the limit, we can replace dH(s) in (1.25) by E. This can be justified by 

transforming the representation (1.25) by a change of variables (letting x = t — s for t fixed) and 

an integration by parts, before applying the renewal theorem, as follows 


Il 
l 
S= 
SS 
ES 
> 
E 
l 
y 
——a 
I 
= 
a 
~ 
l 
a 


The following properties of the joint distribution of (L, y) under FL, are usually much easier to 
work with than the expression (1.27): 


Moments: E[L*] = my+1/ma and Ely*] = mpyi/((k + 1)m1), 
Density of y: y has density f,(s) = (1 — F(s))/ma, 
Joint density: If F has density f, then (L, y) has density 


f(u)/m if0<s<u 


fLylu, s) = { 0 else 


Equivalently, L has density f,(u) = uf(u)/m, and given L = u, y is uniformly distributed 
over the interval [0, u]. 
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Laplace transforms: FY(s) = (1 — F*(s))/(sm1) and F(s) = —(dF*(s)/ds)/ma. 


The fact that the stationary sampled lifetime density, fz(u), is proportional to uf(u), makes 
sense according to the intuition gleaned from the countries example. The lifetimes here play the role 
of countries in that example, and sampling by a particular time (analogous to selecting a country 
by first selecting an individual) results in a factor u proportional to the size of the lifetime. 


1.6.2 Renewal Theory in Discrete Time 


Suppose that the variables T,,7>, ... are integer valued. The process a(t) need only be considered 
for integer values of t. The renewal equation can be written as h = g+ f xh. where h(n) is 
the expected number of renewals that occur at time n, and f and g are the probability mass 
functions corresponding to F and G. The z-transform of h is readily found to be given by H(z) = 


G(z)/(1 — F(z). 
Recall that the greatest common divisor, GCD, of a set of positive integers is the largest integer 
that evenly divides all the integers in the set. For example, GCD{4, 6,12} = 2. 


Proposition 1.6.2 (Blackwell’s Renewal Theorem in discrete time) Suppose GCD{k > 1: f(k) > 
0) =1. Then limy oo h(n) = 1/m,, where my is the k” moment of F. 


The proposition can be used to prove the following fact. If GCD{k > 1: f(k) > 0} = 1, then 
(Ln, Yn) converges in distribution as n — oo, and under the limiting distribution 


PLL =l] =lfi/m and, .Ply a bay a 1b 1s 


_1-F(i-1) 
= 


Ph =i] HE 
and 
ELL] =mp+1/m3 and Ely] = ((m2/2m1)) + 0.5. 


Note that if a renewal occurs at time t, then by definition + is the time between t and the first 
renewal after time t. An alternative definition would take y equal to zero in such a case. Under 
the alternative definition, Ely] = ((m2/2m1)) — 0.5. In fact, this is the same as the mean of the 
backwards residual arrival time for the original definition. In contrast, in continuous time, both the 
forward and backward residual lifetimes have mean m2/2m}. 


1.7 Classification and Convergence of Discrete State Markov Pro- 
cesses 


1.7.1 Examples with finite state space 


Recall that a probability distribution m on S is an equilibrium probability distribution for a time- 
homogeneous Markov process X if m = rH(t) for all t. We shall see in this section that under 
certain natural conditions, the existence of an equilibrium probability distribution is related to 
whether the distribution of X(t) converges as t — oo. Existence of an equilibrium distribution is 
also connected to the mean time needed for X to return to its starting state. To motivate the 
conditions that will be imposed, we begin by considering four examples of finite state processes. 
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Then the relevant definitions are given for finite or countably-infinite state space, and propositions 
regarding convergence are presented. The key role of renewal theory is exposed. 

Example 1.1 Consider the discrete-time Markov process with the one-step probability diagram 
shown in Figure 1.9. Note that the process can’t escape from the set of states Sı = [a,b, c, d,e}, 
so that if the initial state X(0) is in Sı with probability one, then the limiting distribution is 
supported by Sı. Similarly if the initial state X(0) is in S2 = {f,g,h} with probability one, then 
the limiting distribution is supported by S2. Thus, the limiting distribution is not unique for this 
process. The natural way to deal with this problem is to decompose the original problem into two 
problems. That is, consider a Markov process on S41, and then consider a Markov process on Ss. 


1 i 2 
TA O) 
0.5 


a 1 K 
vanas 
G) aa 


Figure 1.9: Example 1.1 


Does the distribution of X (0) necessarily converge if X(0) € Sı with probability one? The 
answer is no. For example, note that if X(0) = a, then X(k) € {a,c,e} for all even values of k, 
whereas X (k) € {b, d} for all odd values of k. That is, ma(k) + r¿(k) + re(k) is one if k is even and 
is zero if k is odd. Therefore, if r¿(0) = 1, then r(k) does not converge as k — oo. 

Basically speaking, the Markov process of this example fails to have a unique limiting distribu- 
tion independent of the initial state for two reasons: (i) the process is not irreducible, and (ii) the 
states are periodic. 

Example 1.2 Consider the two-state, continuous time Markov process with the transition rate 
diagram shown in Figure 1.10 for some positive constants aœ and 8. The generator matrix is given 


Q= 


B 


Figure 1.10: Example 1.2 


by 
—a a 
o| | 
We shall compute H(t) = e%* by solving the Kolmogorov forward equations. The first equation is 
Op11(t 
pnl = —apıı (t) + Ppralt);  pr(0) = 1 
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But pia(t) = 1 — p11 (t), so 


Op11(t) 
Ot 


=—(a+ B)pu(t) +8;  pr(0) =1 
By differentiation we check that this equation has the solution 


t 
pu(t) = enero 4 / e- (+A) gas 
0 


p 
a+ 


(Lae), 


Using the fact pi2(t) = 1 — pii(t) yields pı2(t). The second row of H(t) can be found similarly, 
yielding 


BL 
H(t) = ae e $ po 
atB a+B at+B 


a _a 
+ e (athe | atp are | 


Note that H(0) = I as required, and that 


Thus, for any initial distribution 7(0), 


as ay at A 
am at) = oe EES (a +8 a+p 


Ji 


The rate of convergence is exponential, with rate parameter a+ 6, which happens to be the nonzero 
eigenvalue of Q. Note that the limiting distribution is the unique probability distribution satisfying 
TQ = 0. The periodicity problem of Example 1.1 does not arise for continuous-time processes. 

Example 1.3 Consider the continuous-time Markov process with the transition rate diagram 
in Figure 1.11. The Q matrix is the block-diagonal matrix given by 


B B 


Figure 1.11: Example 1.3 


—a a 0 0 

Bo -£ 0 0 

Q 0 0 -a a 
o 0 8 - 


This process is not irreducible, but rather the transition rate diagram can be decomposed into two 
parts, each equivalent to the diagram for Example 1.1. The equilibrium probability distributions 
are the probability distributions of the form 7 = Og, Ap Cl NE (1 — M5), Where A is 
the probability placed on the subset {1,2}. 
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Figure 1.12: Example 1.4 


Example 1.4 Consider the discrete-time Markov process with the transition probability dia- 
gram in Figure 1.12. The one-step transition probability matrix P is given by 


0 1 0 
P=;0 01 
1 0 0 


Solving the equation m = TP we find there is a unique equilibrium probability vector, namely 
m = (4,4,4). On the other hand, if 7(0) = (1,0,0), then 
(1,0,0) ifk =0 mod 3 
a(k) =7(0)P* =< (0,1,0) ifk=1mod3 
(0,0,1) if k =2 mod 3 


Therefore, 7(k) does not converge as k — oo. 

Let X be a discrete-time, or pure-jump continuous-time, Markov process on the countable state 
space S. Suppose X is time-homogeneous. The process is said to be irreducible if for all i,j € S, 
there exists s > 0 so that p;;(s) > 0. 


1.7.2 Discrete Time 


Clearly a probability distribution m is an equilibrium probability distribution for a discrete-time 
time-homogeneous Markov process X if and only if 7 = TP. The period of a state 1 is defined 
to be GCD{k > 0 : pulk) > 0}. The set {k > 0: pulk) > 0} is closed under addition, which 
by elementary algebra ? implies that the set contains all sufficiently large integer multiples of the 
period. The Markov process is called aperiodic if the period of all the states is one. 


Proposition 1.7.1 If X is irreducible, all states have the same period. 


Proof. Let i and j be two states. By irreducibility, there are integers kı and kz so that p;j(k1) > 0 
and pji(k2) > 0. For any integer n, pii(n + kı + k2) > pij(kı)pjj(n)pji(k2), so the set {k > 0: 
pilk) > 0] contains the set {k > 0: pj;(k) > 0} translated up by kı + k2. Thus the period of i is 
less than or equal to the period of j. Since i and j were arbitrary states, the proposition follows. Mi 


For a fixed state i, define 7 = min{k > 1 : X(k) = i}, where we adopt the convention 
that the minimum of an empty set of numbers is +00. Let Mi = Elr¡[X(0) = ij. If Plr, < 
+oo| X(0) = i] < 1, then state i is called transient (and by convention, M; = +00). Otherwise 
P[r; < +o00|X(0) = i] = 1, and 7 is then said to be positive — recurrent if M; < +00 and to be 
null — recurrent if M; = +00. 


see Euclidean algorithm, Chinese remainder theorem, Bezout theorem 
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Proposition 1.7.2 Suppose X is irreducible and aperiodic. 
(a) All states are transient, or all are positive-recurrent, or all are null-recurrent. 


(b) For any initial distribution (0), limyoo m(t) =1/M;, with the understanding that the limit 
is zero if M; = +00. 


(c) An equilibrium probability distribution m exists if and only if all states are positive-recurrent. 


(d) If it exists, the equilibrium probability distribution m is given by m; =1/M;. (In particular, if 
it exists, the equilibrium probability distribution is unique). 


Proof. (a) Suppose state i is recurrent. Given X(0) = i, after leaving i the process returns to 
state į at time 7;. The process during the time interval {0,...,7;} is the first excursion of X from 
state 0. From time 7; onward, the process behaves just as it did initially. Thus there is a second 
excursion from i, third excursion from i, and so on. Let Ty for k > 1 denote the length of the kth 
excursion. Then the 7;,’s are independent, and each has the same distribution as 71 = 7;. Let j be 
another state and let e denote the probability that X visits state j during one excursion from i. 
Since X is irreducible, e > 0. The excursions are independent, so state j is visited during the kth 
excursion with probability e, independently of whether j was visited in earlier excursions. Thus, the 
number of excursions needed until state j is reached has the geometric distribution with parameter 
e, which has mean 1/e. In particular, state j is eventually visited with probability one. After j 
is visited the process eventually returns to state 7, and then within an average of 1/e additional 
excursions, it will return to state j again. Thus, state j is also recurrent. Hence, if one state is 
recurrent, all states are recurrent. 

The same argument shows that if i is positive-recurrent, then j is positive-recurrent. Given 
X(0) = i, the mean time needed for the process to visit j and then return to i is M;/e, since on 
average 1/e excursions of mean length M; are needed. Thus, the mean time to hit j starting from 
i, and the mean time to hit 7 starting from j, are both finite. Thus, 7 is positive-recurrent. Hence, 
if one state is positive-recurrent, all states are positive-recurrent. 

(b) Part (b) of the proposition follows immediately from the renewal theorem for discrete time. 
The first lifetime for the renewal process is the time needed to first reach state 1, and the subsequent 
renewal times are the lengths of the excursions from state i. The probability 7;(t) is the probability 
that there is a renewal event at time t, which by the renewal theorem converges to 1/M;. 

(c) Suppose all states are positive-recurrent. By the law of large numbers, for any state j, the 
long run fraction of time the process is in state j is 1/M,; with probability one. Similarly, for any 
states i and j, the long run fraction of time the process is in state j is yi¡/M;, where yi; is the 
mean number of visits to j in an excursion from i. Therefore 1/M; = yi¿/M;. This implies that 
yo, 1/M; = 1. That is, m defined by 7; = 1/M; is a probability distribution. The convergence for 
each 7 separately given in part (b), together with the fact that 7 is a probability distribution, imply 
that 5°, [m;(t) — mi| — 0. Thus, taking s to infinity in the equation r(s)H(t) = a(s + t) yields 
TH (t) = 7, so that m is an equilibrium probability distribution. 

Conversely, if there is an equilibrium probability distribution 7, consider running the process 
with initial state m. Then z(t) = 7 for all t. So by part (b), for any state i, 7; = 1/M;. Taking a 
state į such that m; > 0, it follows that M; < oo. So state i is positive-recurrent. By part (a), all 
states are positive-recurrent. 

(d) Part (d) was proved in the course of proving part (c). a 
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We conclude this section by describing a technique to establish a rate of convergence to the 
equilibrium distribution for finite-state Markov processes. Define 6(P) for a one-step transition 
probability matrix P by 


5(P) = min Y py A Pri 
( ) mip LP Pkj»> 


where a Ab = min{a, bj. The number 6(P) is known as Dobrushin’s coefficient of ergodicity. Since 
a+b—2(aAb) = la — b| for a,b > 0, we also have 


1 — 26(P) = mi ii — Phil. 
ô( ) mp 2 ley Prjl 


Let [ul] for a vector u denote the Lı norm: ||| =>", |1u5!. 


Proposition 1.7.3 For any probability vectors n and o, [rP=0Pl < (1—46(P))||7— ||. Further- 
more, if 9(P) > 0 then there is a unique equilibrium distribution T, and for any other probability 
distribution T on S, ||P! — || < 2(1 — 6(P))!. 


Proof. Let #; = m; — m;i Ao; and 6; = o; — mi A oj. Clearly ||r — o|| = ||# — || = 217 [| = 2/|6||. 
Furthermore, 


|tP —oP|| = ||\7P-—GP| 


= Y 1) mPi — SS Psl 
i k 


j 


(IF OI ice Pig — Pes) 


j ik 


< (YI) tice SO Pig — Pas] 
i,k j 
< [Zl — 26(P)) = [lr — al — 4(P)), 


which proves the first part of the proposition. Iterating the inequality just proved yields that 
aP! — oP!|| < (1-0) lr — ol] < 2(1 — 0). (1.28) 


This inequality for o = nP” yields that |r P! — nP!'+”]| < 2(1 — ô)!. Thus the sequence TP! is a 
Cauchy sequence and has a limit 7, and nP = 7°. Finally, taking o in (1.28) equal to 7% 
yields the last part of the proposition. 


Proposition 1.7.3 typically does not yield the exact asymptotic rate that ||! — “|| tends to 
zero. The asymptotic behavior can be investigated by computing (J — zP)~', and then matching 
powers of z in the identity (I — zP)! = 22” P”. 


1.7.3 Continuous Time 


Define, for i € S, 72 = min{t > 0: X(t) # i}, and 7, = min{t > 72 : X(t) = i}. Thus, if 
X(0) = i, 7 is the first time the process returns to state i, with the exception that 7; = +00 
if the process never returns to state i. The following definitions are the same as when X is a 
discrete-time process. Let M; = Efri|X (0) = i]. If P[r, < +00] < 1, then state i is called transient. 
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Otherwise P[7, < +00] = 1, and 7 is then said to be positive — recurrent if Mi < +00 and to be 
null — recurrent if M; = +00. 

The following propositions are analogous to those for discrete-time Markov processes. Proofs 
can be found in [1, 32]. 


Proposition 1.7.4 Suppose X is irreducible. 
(a) All states are transient, or all are positive-recurrent, or all are null-recurrent. 


(b) For any initial distribution 7(0), limp++oo Tilt) = 1/(—qi: Mi), with the understanding that the 
limit is zero if M; = +00. 


Proposition 1.7.5 Suppose X is irreducible and nonexplosive. 
(a) A probability distribution m is an equilibrium distribution if and only if TQ = 0. 
(b) An equilibrium probability distribution exists if and only if all states are positive-recurrent. 


(c) If all states are positive-recurrent, the equilibrium probability distribution is given by m = 
1/(—qiMi). (In particular, if it exists, the equilibrium probability distribution is unique). 


Once again, the renewal theorem leads to an understanding of the convergence result of part 
(b) in Proposition 1.7.4. Let Tẹ denote the amount of time between the kth and k + 1th visits to 
state 1. Then the variables Th : k > 0 are independent and identically distributed, and hence the 
sequence of visiting times is a renewal sequence. Let H(t) denote the expected number of visits to 
state i during the interval [0, t], so that H is the cumulative mean renewal function for the sequence 
T,,T3,.... As above, note that M; = E[T;]. 

The renewal theorem is applied next to show that 

Jim P[X(t) = = 1/(qMi). (1.29) 

Indeed, note that the definition of H, the fact that holding times for state 7 are exponentially 
distributed with parameter qii, integration by parts, a change of variable, and the renewal theorem, 
imply that 


PA =a) = PM => + + fr PIX , last renewal before t) € ds] 


= P[X(0) = dJexp(qit) + Cual exp(qu(t — s))dH(s) 

| explault = ad LO ~ H(s)}as 
exp( 

0 


I 
i 


= / ou da Maw 1 Esto) 


= P[X(0) = 1] exp(qut) + 


= P[X(0) =1] exp(qut) + | exp(qieu)(—gu){H(t) — H(t — u) }du 


This establishes (1.29). 
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An immediate consequence of (1.29) is that if there is an equilibrium distribution 7, then the 
Markov process is positive recurrent and the equilibrium distribution is given by 


Conversely, if the process is positive recurrent, then there is an equilibrium probability distribution 
and it is given by (1.30). 

Part (a) of Proposition 1.7.5 is suggested by the Kolmogorov forward equation Ox(t)/Ot = 
m(t)Q. The assumption that X be nonexplosive is needed (per homework problem), but the follow- 
ing proposition shows that the Markov processes encountered in most applications are nonexplosive. 


Proposition 1.7.6 Suppose X is irreducible. Fix a state io and for k > 1 let Sy, denote the set of 
states reachable from i in k jumps. Suppose for each k > 1 there is a constants yp so that the jump 
intensities on Sp are bounded by yy, that is, suppose —qii < Yk for i € Sk. If NY z E = +00, then 
the process X is nonexplosive. 


The proposition is proved after the following lemma is proved: 


Lemma 1.7.7 Suppose S is a random variable such that E[S] > 1 and S is the sum of finitely 
many independent exponentially distributed random variables (with possibly different means). Then 
P[S > 3] >]. 


Proof. Let S = Ti +---+7),, such that the T; are independent, exponentially distributed, and 
suppose E[S] > 1. If E[T;] > 3 for some j then P[S > 5] > P[T; > 3] > e! > 3. So it remains 
to prove the lemma in case E[T;] < 3 for all i. But then Var(Z;) = E[T;]? < FE[T;]. Therefore, 


Var(S) < eo) Thus, by the Chebychev inequality, P[S < 5] < P[|S — E[S]| > zal < wes < 


TEIS] < 3; The lemma is proved. E 


Proof. Let X7 denote the jump process and To, Tı,... the sequence of holding times of X. 
Select an increasing sequence of integers ko = 0 < kı < ka < --- so that a > > 1 for all 
¿> 1. Fix any particular sample path (u7(n)) for the jump process with x7(0) = io. Conditioned 
on the process X” following this sample path, the holding times are independent, exponentially 
distributed, and E[Tp] > z for all k > 0. Thus, if S; is the random variable S; = Seay 
it follows from Lemma 1.7.7 that P[S; > 2] = i for all i. Thus, for a suitable threshold 7; > z, 
the random variable O, = I{s;>7;} is a Be(4) random variable, S; > Si for all i, and the ©;’s 
are independent. Thus, ya Pr Tk > sa O;, and by the law of large number, yy O; > oo as 
j — œ. Thus, the sum of the holding times is a.s. infinite, so X is nonexplosive. E 


1.8 Classification of Birth-Death Processes 


The classification of birth-death processes is relatively simple. Recall that a birth-death process 
with birth rates (Az : k > 0) and death rates (up : k > 1) is a pure-jump Markov process with state 
space equal to the set of nonnegative integers Z+} (with the appended state A, which is reached if 
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and only if infinitely many jumps occur in finite time), and generator matrix Q = (qij : i,j > 0) 
given by 

Ai if j=i+1 

Mi if j=i-1 


qij =§ -Ai Mm if j=i21 (1.31) 
-\; if j=i=l 
0 else. 


Suppose that the birth and death rates are all strictly positive. Then the process is irreducible. Let 
bin denote the probability that, given the process starts in state 1, it reaches state n without first 
reaching state 0. Clearly bo, = 0 and bnn = 1. Fix ¿ with 1 <i < n — 1, and derive an expression 
for bin by first conditioning on the state reached upon the first jump of the process, starting from 
state i. By the analysis of jump probabilities, the probability the first jump is up is A¿/(A; + pi) 
and the probability the first jump is down is p;/(Ai + ui). Thus, 


i= Ài b; , Hi 
nO tu A 
which can be rewritten as [i (Din — Di) = AiO = bin). In particular, bən — bin = binpa/A 
and b3n — ban = binp1p2/(A142), and so on, which upon summing yields the expression 


bi—1, n, 


with the convention that the i = 0 term in the sum is zero. Finally, the condition bnn = 1 yields 


the solution é 


ars papa. pe ` 
1=0 Ay AQ. AG 


Note that bın is the probability, for initial state 1, of the event Bn that state n is reached without 
an earlier visit to state 0. Since Bn+1 C B for all n > 1, 


bin = (1.32) 


P[An>1Bn|X (0) = 1] = lim bin = 1/S2 (1.33) 


where S2 = X; o M1p2 ++ /A1 A... Ai, with the understanding that the i = 0 term in the sum 
defining S2 is one. Due to the definition of pure jump processes used, whenever X visits a state 
in S the number of jumps up until that time is finite. Thus, on the event Nn>1Bn, state zero is 
never reached. Conversely, if state zero is never reached, then either the process remains bounded 
(which has probability zero) or Nn>1Bn is true. Thus, Plzero is never reached|X(0) = 1] = 1/52. 
Consequently, X is recurrent if and only if S2 = oo. 

Next, investigate the case S2 = oo. Then the process is recurrent, so that in particular explosion 
has probability zero, so that the process is positive recurrent if and only if TQ = 0. 

Now TQ = 0 if and only if flow balance holds for any state k: 


(Ak + Mk) Tr = Ar1T-1 + Hk+1Tk+1- (1.34) 


Equivalently, flow balance must hold for all sets of the form {0,...,2—1}, yielding that m,-1An—1 = 
TnAn, OF Tn = Todo ---An—1/(111 --- Hn). Thus, a probability distribution 7 with TQ = 0 exists if 
and only if S1 < +00, where 


ap er, ME 
= Sa, (1.35) 
n=0 psi 


with the understanding that the i = 0 term in the sum defining Sı is one. Still assuming S2 = oo, 
such a probability distribution exists if and only if the Markov process is positive recurrent. 
In summary, the following proposition is proved. 


Proposition 1.8.1 If S2 < +00 then all the states in Z} are transient, and the process may 
be explosive. If S2 = +00 and Si = +00 then all states are null-recurrent. If S2 = +00 and 
Sı < +00 then all states are positive-recurrent and the equilibrium probability distribution is given 
by Tn = Do Tai An=1)/(5141 ee ln). 


Discrete-time birth-death processes have a similar characterization. They are discrete-time, 
time-homogeneous Markov processes with state space equal to the set of nonnegative integers. Let 
nonnegative birth probabilities (Az : k > 0) and death probabilities (uz: k > 1) satisfy Ao < 1, and 
Ak + Uk < 1 for k > 1. The one-step transition probability matrix P = (pi; : i, j > 0) is given by 


Ai if j=i+1 
li if j=i-1 
Pij = 1 — Ài — hi if =121 (1.36) 
1-\ if j=i=l 
0 else. 


Implicit in the specification of P is that births and deaths can’t happen simultaneously. If the birth 
and death probabilities are strictly positive, Proposition 1.8.1 holds as before, with the exception 
that the discrete-time process cannot be explosive. 3 


1.9 Time Averages vs. Statistical Averages 


Let X be a positive recurrent, irreducible, time-homogeneous Markov process with equilibrium 
probability distribution 7. To be definite, suppose X is a continuous-time process, with pure-jump 
sample paths and generator matrix Q. The results of this section apply with minor modifica- 
tions to the discrete-time setting as well. Above it is shown that renewal theory implies that 
limp soo milt) = mi = 1/(—qiiMi), where M; is the mean “cycle time” of state i. A related con- 
sideration is convergence of the empirical distribution of the Markov process, where the empirical 
distribution is the distribution observed over a (usually large) time interval. 
For a fixed state i, the fraction of time the process spends in state ¿i during [0, t] is 


1 t 
E f L{x(s)=i} ds 


Let Tp denote the time that the process is first in state i, and let Tk for k > 1 denote the time 
that the process jumps to state 1 for the kth time after Tp. The cycle times Ty11 — Tk, k > 0 are 
independent and identically distributed, with mean M;. Therefore, by the law of large numbers, 
with probability one, 


k— 


; ; 1 
Jim Tk/k = lim ¡Ea -= T;) (1.37) 
= Mi; (1.38) 


3If in addition A; + u; = 1 for all ¿, then the discrete time process has period 2. 
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Furthermore, during the kth cycle interval [T;, Tr. 1), the amount of time spent by the process 
in state ¿ is exponentially distributed with mean —1/q;;, and the time spent in the state during 
disjoint cycles is independent. Thus, with probability one, 


i 1 te i k-1 Tipi 

jim [ Lx (s)=i)U = m2), Lx (s)=i) ds (1.39) 

Tı 
= Ef Ir x(s)=1} 45] (1.40) 

To 
= —1/qü (1.41) 

Combining these two observations yields that 
ye AA 

Jm E a Lix(s)= ¡pls = =1/(-q4Mi) = Ti (1.42) 


with probability one. In short, the limit (1.42) is expected, because the process spends on average 
—1/q;; time units in state i per cycle from state i, and the cycle rate is 1/M;. Of course, since state 
i is arbitrary, if j is any other state then 


1 
im = Pre )=jyds = 1/(—qj; Mj) = Tj (1.43) 
Do t 
By considering how the time in state j is distributed among the cycles from state 1, it follows that 
the mean time spent in state j per cycle from state i is Minj. 
So for any nonnegative function ¢ on S, 


Tk 
e T ja iwe Ae 


tooo t 


= Y oir; (1.44) 


Finally, if ¢ is a function of S such that either > jes 6+(7)™j < œ or $ jes 6-(J) 7; < 00, then 
since (1.44) holds for both ¢4 and ¢_, it must hold for ¢ itself. 


1.10 Queueing Systems, M/M/1 Queue and Little’s Law 


Some basic terminology of queueing theory will now be explained. A simple type of queueing system 
is pictured in Figure 1.13. Notice that the system is comprised of a queue and a server. Ordinarily 
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Figure 1.13: A single server queueing system 


whenever the system is not empty, there is a customer in the server, and any other customers in 
the system are waiting in the queue. When the service of a customer is complete it departs from 
the server and then another customer from the queue, if any, immediately enters the server. The 
choice of which customer to be served next depends on the service discipline. Common service 
disciplines are first-come first-served (FCFS) in which customers are served in the order of their 
arrival, or last-come first-served (LCFS) in which the customer that arrived most recently is served 
next. Some of the more complicated service disciplines involve priority classes, or the notion of 
“processor sharing” in which all customers present in the system receive equal attention from the 
server. 

Often models of queueing systems involve a stochastic description. For example, given positive 
parameters À and u, we may declare that the arrival process is a Poisson process with rate A, 
and that the service times of the customers are independent and exponentially distributed with 
parameter u. Many queueing systems are given labels of the form A/B/s, where “A” is chosen to 
denote the type of arrival process, “B” is used to denote the type of departure process, and s is 
the number of servers in the system. In particular, the system just described is called an M/M/1 
queueing system, so-named because the arrival process is memoryless (i.e. a Poisson arrival process), 
the service times are memoryless (i.e. are exponentially distributed), and there is a single server. 
Other labels for queueing systems have a fourth descriptor and thus have the form A/B/s/b, where 
b denotes the maximum number of customers that can be in the system. Thus, an M/M/1 system 
is also an M/M/1/oco system, because there is no finite bound on the number of customers in the 
system. 

A second way to specify an M/M/1 queueing system with parameters À and p is to let A(t) and 
D(t) be independent Poisson processes with rates A and y respectively. Process A marks customer 
arrival times and process D marks potential customer departure times. The number of customers in 
the system, starting from some initial value N(0), evolves as follows. Each time there is a jump of 
A, a customer arrives to the system. Each time there is a jump of D, there is a potential departure, 
meaning that if there is a customer in the server at the time of the jump then the customer departs. 
If a potential departure occurs when the system is empty then the potential departure has no effect 
on the system. The number of customers in the system N can thus be expressed as 


N(t) = N(0) + A(t) + i Ign (s—)>1}dD(s) 


It is easy to verify that the resulting process N is Markov, which leads to the third specification of 
an M/M/1 queueing system. 
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Figure 1.14: A single server queueing system 


A third way to specify an M/M/1 queuing system is that the number of customers in the system 
N (t) is a birth-death process with A, = A and ug = y for all k, for some parameters À and u. Let 
p = X/p. Using the classification criteria derived for birth-death processes, it is easy to see that 
the system is recurrent if and only if p < 1, and that it is positive recurrent if and only if p < 1. 
Moreover, if p < 1 then the equilibrium distribution for the number of customers in the system is 
given by 7, = (1 — p)p* for k > 0. This is the geometric distribution with zero as a possible value, 
and with mean 


N=) km =(1- pp» p kal ay Ge 
k=0 k=1 


The probability the server is busy, which is also the mean number of customers in the server, is 
1— 7o = p. The mean number of customers in the queue is thus given by p/(1 — p) — p = p?/(1— p). 
This third specification is the most commonly used way to define an M/M/1 queueing process. 

Since the M/M/1 process N(t) is positive recurrent, the Markov ergodic convergence theorem 
implies that the statistical averages just computed, such as N, are also equal to the limit of the 
time-averaged number of customers in the system as the averaging interval tends to infinity. 

An important performance measure for a queueing system is the mean time spent in the system 
or the mean time spent in the queue. Littles’ law, described next, is a quite general and useful 
relationship that aids in computing mean transit time. 

Little’s law can be applied in a great variety of circumstances involving flow through a system 
with delay. In the context of queueing systems we speak of a flow of customers, but the same 
principle applies to a flow of water through a pipe. Little’s law is that AT = N where A is the 
mean flow rate, T is the mean delay in the system, and N is the mean content of the queue. For 
example, if water flows through a pipe with volume one cubic meter at the rate of two cubic meters 
per minute, then the mean time (averaged over all drops of water) that water spends in the pipe is 
T = N/A= 1/2 minute. This is clear if water flows through the pipe without mixing, for then the 
transit time of each drop of water is 1/2 minute. However, mixing within the pipe does not effect 
the average transit time. 

Little’s law is actually a set of results, each with somewhat different mathematical assumptions. 
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The following version is quite general. Figure 1.14 pictures the cumulative number of arrivals (a(t)) 
and the cumulative number of departures (d(t)) versus time, for a queueing system assumed to be 
initially empty. Note that the number of customers in the system at any time s is given by the 
difference N(s) = a(s) — 6(s), which is the vertical distance between the arrival and departure 
graphs in the figure. On the other hand, assuming that customers are served in first-come first- 
served order, the horizontal distance between the graphs gives the times in system for the customers. 
Given a (usually large) t > 0, let y denote the area of the region between the two graphs over the 
interval [0,t]. This is the shaded region indicated in the figure. It is then natural to define the 
time-averaged values of arrival rate and system content as 


= Sas J he ae 
At = a(t)/t and M=; f N(s)ds = w/t 
0 


Finally, the average, over the a(t) customers that arrive during the interval (0, t], of the time spent 
in the system up to time t, is given by 


T; = v/a(t). 
Once these definitions are accepted, we have the obvious simple relation: 
Ne= ATi (1.45) 


Consider next (1.45) in the limit as t — oo. If any two of the three variables in (1.45) converge to 
finite positive quantities, then so does the third variable, and in the limit Little’s law is obtained. 
For example, the number of customers in an M/M/1 queue is a positive recurrent Markov 
process so that 
Jim Ne =N =p/(1- p) 


where calculation of the statistical mean N was previously discussed. Also, by the weak renewal 
theorem or the law of large numbers applied to interarrival times, we have that the Poisson arrival 
process for an M/M/1 queue satisfies lim, Az = A with probability one. Thus, 

inne we N 

nen ue T pb A 
In this sense, the average waiting time in an M/M/1 system is 1/(u—A). The average time in service 
is 1/p (this follows from the third description of an M/M/1 queue, or also from Little’s law applied 
to the server alone) so that the average waiting time in queue is given by W = 1/(u — A) — 1/u = 
p/(u — A). This final result also follows from Little’s law applied to the queue alone. 


1.11 Mean Arrival Rate, Distributions Seen by Arrivals, and PASTA 


The mean arrival rate for the M/M/1 system is A, the parameter of the Poisson arrival process. 
However for some queueing systems the arrival rate depends on the number of customers in the 
system. In such cases the mean arrival rate is still typically meaningful, and it can be used in 
Little’s law. 

Suppose the number of customers in a queuing system is modeled by a birth death process 
with arrival rates (Az) and departure rates (upk). Suppose in addition that the process is positive 
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recurrent. Intuitively, the process spends a fraction of time mẹ in state k and while in state k the 
arrival rate is Az. Therefore, the average arrival rate is 


A= oS T KAR 
k=0 
Similarly the average departure rate is 
OO 
H= y TkHk 
k=1 


and of course À = Zi because both are equal to the throughput of the system. 

Often the distribution of a system at particular system-related sampling times are more impor- 
tant than the distribution in equilibrium. For example, the distribution seen by arriving customers 
may be the most relevant distribution, as far as the customers are concerned. If the arrival rate 
depends on the number of customers in the system then the distribution seen by arrivals need not 
be the same as the equilibrium distribution. Intuitively, 7A, is the long-term frequency of arrivals 
which occur when there are k customers in the system, so that the fraction of customers that see 
k customers in the system upon arrival is given by 

_ TRA 
Tk o 
The following is an example of a system with variable arrival rate. 
Example 1.5 (Single-server, discouraged arrivals) 
Suppose A. = a/(k +1) and ug = y for all k, where u and a are positive constants. Then 


s=y' oe =oo and 5=) q. = pl ) <œ 
k=0 k=0 


so that the number of customers in the system is a positive recurrent Markov process, with no 
additional restrictions on a and u. Moreover, the equilibrium probability distribution is given by 


Ty = (a/u) exp(—a/p)/k!, which is the Poisson distribution with mean N = a/u. The mean 
arrival rate is 


Q 


©). (146) 


= pep (exp) — 1) = p(1 — exp( 


This expression derived for A is clearly equal to m, because the departure rate is u with probability 
1 — mp and zero otherwise. The distribution of the number of customers in the system seen by 
arrivals, (rg) is given by 


k+1 z 
ee CRA a E e et 


(k+1DA (k+ 1)1(1 — exp(—a/p)) 
which in words can be described as the result of removing the probability mass at zero in the 
Poisson distribution, shifting the distribution down by one, and then renormalizing. The mean 
number of customers in the queue seen by a typical arrival is therefore (a/u — 1)/(1 — exp(-Q/p)). 
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This mean is somewhat less than N because, roughly speaking, the customer arrival rate is higher 
when the system is more lightly loaded. 

The equivalence of time-averages and statistical averages for computing the mean arrival rate 
and the distribution seen by arrivals can be shown by application of ergodic properties of the 
processes involved. The associated formal approach is described next, in slightly more generality. 
Let X denote an irreducible, positive-recurrent pure-jump Markov process. If the process makes a 
jump from state i to state j at time t, say that a transition of type (i, j) occurs. The sequence of 
transitions of X forms a new Markov process, Y. The process Y is a discrete-time Markov process 
with state space {(i, j) E S x S : qij > 0), and it can be described in terms of the jump process for 
X, by Y(k) = (X7(k — 1), X7(k)) for k > 0. (Let X7(—1) be defined arbitrarily.) 

The one-step transition probability matrix of the jump process X” is given by ni = ij /(— qii), 
and X7 is recurrent because X is recurrent. Its equilibrium distribution 77 (if it exists) is propor- 
tional to —1;q;; (see problem set 1), and X7 is positive recurrent if and only if this distribution can 
be normalized to make a probability distribution, i.e. if and only if R = —) 7, Tiqii < 00. Assume 
for simplicity that X” is positive recurrent. Then Ti = —Tiqü/R is the equilibrium probability 
distribution of X7. Furthermore, Y is positive recurrent and its equilibrium distribution is given 
by 


Ti, = Ti pi, 
— TMi lij 
R —qü 

— Midi 

© R 


Since limiting time averages equal statistical averages for Y, 
lim (number of first n transitions of X that are type (i, j))/n = miqij/R 
n—00 


with probability one. Therefore, if A C S x S, and if i € A, then 


. number of first n transitions of X that are type (i, 7) Tiqij 
lim — - - = (1.47) 
n>œ number of first n transitions of X with type in A EA Tit qu! 


We remark (1.47) is still true if the assumption R < oo is replaced by the weaker assumption that 
the denominator on the righthand side of (1.47) is finite. In case R = co the statement is then an 
instance of a ratio limit theorem for recurrent (not necessarily positive recurrent) Markov processes. 

To apply this setup to the special case of a queueing system in which the number of customers 
in the system is a Markov birth-death processes, let the set A be the set of transitions of the form 
(i i +1). Then deduce that the fraction of the first n arrivals that see i customers in the system 
upon arrival converges to 7A; / >> F TjAj with probability one. 

Note that if A; = A for all 2, then A= A and t =r. The condition A; = A also implies that the 
arrival process is Poisson. This situation is called “Poisson Arrivals See Time Averages” (PASTA). 


1.12 More Examples of Queueing Systems Modeled as Markov 
Birth-Death Processes 


For each of the four examples of this section it is assumed that new customers are offered to the 
system according to a Poisson process with rate A, so that the PASTA property holds. Also, when 
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there are k customers in the system then the service rate is uz for some given numbers pz. The 
number of customers in the system is a Markov birth-death process with A, = A for all k. Since 
the number of transitions of the process up to any given time t is at most twice the number of 
customers that arrived by time t, the Markov process is not explosive. Therefore the process is 
positive recurrent if and only if S4 is finite, where 


p1p2..- Hk 


Special cases of this example are presented in the next four examples. 

Example 1.6 (M/M/m systems) An M/M/m queueing system consists of a single queue and 
m servers. The arrival process is Poisson with some rate A and the customer service times are 
independent and exponentially distributed with mean u for some u > 0. The total number of 
customers in the system is a birth-death process with uw, = umin(k, m). Let p = A/(mp). Since 
Hk = my for all k large enough it is easy to check that the process is positive recurrent if and only 
if p < 1. Assume now that p < 1. Then the equilibrium distribution is given by 


0/1" 
Sik! 


Tm+j = Tmp! for 7 >1 


for0<k<m 


where S1 is chosen to make the probabilities sum to one (use the fact 1+p+p?...=1/(1— p)): 


m-1 A k A m 
= ES e ) a e D 


k=0 


An arriving customer must join the queue (rather that go directly to a server) if and only if the 
system has m or more customers in it. By the PASTA property, this is the same as the equilibrium 
probability of having m or more customers in the system: 


OO 
Po = Y mtj = Tm/(1 — p) 
j=0 


This formula is called the Erlang C formula for probability of queueing. 

Example 1.7 (M/M/m/m systems) An M/M/m/m queueing system consists of m servers. 
The arrival process is Poisson with some rate À and the customer service times are independent 
and exponentially distributed with mean p for some u > 0. Since there is no queue, if a customer 
arrives when there are already m customers in the system, then the arrival is blocked and cleared 
from the system. The total number of customers in the system is a birth death process, but with 
the state space reduced to {0,1,...,m}, and with ug = ku for 1 < k < m. The unique equilibrium 
distribution is given by 

(A/u)* 


Sık! 
where S1 is chosen to make the probabilities sum to one. 

An arriving customer is blocked and cleared from the system if and only if the system already 
has m customers in it. By the PASTA property, this is the same as the equilibrium probability of 


Tk = for0<k<m 
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having m customers in the system: 


This formula is called the Eriang B formula for probability of blocking. 

Example 1.8 (A system with a discouraged server) The number of customers in this system 
is a birth-death process with constant birth rate A and death rates iy = 1/k. It is is easy to 
check that all states are transient for any positive value of A (to verify this it suffices to check that 
S2 < co). It is not difficult to show that N(t) converges to +00 with probability one as t => oo. 

Example 1.9 (A barely stable system) The number of customers in this system is a birth-death 
process with constant birth rate A and death rates uy = ts for all k > 1. Since the departure 
rates are barely larger than the arrival rates, this system is near the borderline between recurrence 
and transience. However, we see that 


oO 


1 
= ae 
k=0 


so that N(t) is positive recurrent with equilibrium distribution 7 = 1/(91(1 + k?)). Note that the 
mean number of customers in the system is 


N= 3 k/(S1(1 + k?)) = 00 
k=0 


By Little’s law the mean time customers spend in the system is also infinity. It is debatable whether 
this system should be thought of as “stable” even though all states are positive recurrent and all 
waiting times are finite with probability one. 


1.13 Method of Phases and Quasi Birth-Death Processes 


The method of phases in a broad sense refers to approximating general probability distributions by 
sums and mixtures of exponential distributions. Such distributions can be obtained as the hitting 
time of a target state of a finite state Markov process. The state of the Markov process at a given 
time represents the “phase” of the distribution. The method permits Markov processes to model 
systems in which the original distributions are not exponential. 

To illustrate the method here we consider the class of random processes called quasi birth and 
death (QBD) processes. These processes generalize M/M/1 queueing models in that the arrival and 
departure rates depend on the state of a phase process. (For more background, see M.F. Neuts, 
Matriz-Geometric Solutions in Stochastic Models — An Algorithmic Approach, The Johns Hopkins 
University Press, Baltimore, MD, 1981 and B. Hajek, Birth-and-Death Processes on the Integers 
with Phases and General Boundaries, Journal of Applied Probability, pp. 488-499, 1982. 

Let k be a positive integer, denoting the number of possible phases. A QBD process is a time- 
homogeneous Markov process. The state space of the process is the set S equal to ((1,0) : 1 > 
1,1 <0 < k}. We call l the level and @ the phase of the state (1,0). Here we shall restrict our 
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attention to continuous time processes. Let Ap, 41,42, and Ago denote k x k matrices so that the 
matrix Q* defined by 


Ao A2 
Ao A, Ag 
ka Ao Ai Az 
Q* = Ae ae (1.48) 


is a generator matrix for the state space S. Equivalently, the matrices Ay and Ag have nonnegative 
entries, the matrices A; and Ago have nonnegative off-diagonal entries and nonpositive diagonal 
entries, the matrix Q defined by Q = Ap + 41 + Ag and the matrix Agp + A2 both have row sums 
equal to zero. 

Suppose that Q* and Q are both irreducible. Define 7 to be the equilibrium probability vector 
for Q, so that TQ = 0 and set u = mAge — 7 Age where e is the k vector of all ones. Then all states 
are positive recurrent for Q* if and only if y < 0. If y < 0 then the equilibrium distribution for Q* 
is given by (£o, 21,...) where 


To = T0/C, Tk = roRf, c= mo(I — R) de, R = AB (1.49) 


where 7 is the equilibrium probability distribution for a k-state Markov process with generator 
matrix Ago + 401B410, and B is the minimal positive solution to 


B = -(A1 + A2BAp) (1.50) 


An effective numerical procedure for finding B is to iterate equation (1.50). In other words, set 

B©) to be the matrix of all zeros, and sequentially compute B by B® = —(A,+ A,B) Ap)”. 
Exercises 
1. Let (Li, ©+) denote the QBD process. Under what conditions on Ao, A1,A2 and Aoo is the 
process (O¿) a Markov process? Prove your answer. 
2. Consider the following variation of an M/M/1 queue. There is a two state Markov process with 
states 1 and 2 and transition rates q19 = 1 and q21 = 2. The two state process is a phase process for 
the queue. When the phase is 1 there are no arrivals to the queue, and when the phase is 2 arrivals 
occur at an average rate of 2 customers per second, according to a Poisson process. Service times of 
all customers are independent and exponentially distributed with mean one. Suppose a QBD model 
is used. What values of the matrices Ag, 41, A2 and Apo should be used? Find (numerical solution 
OK) B,R, and the equilibrium distribution (%/)1>0. Finally, describe the equilibrium probability 
distribution of the number of customers in the system. (Solution: 


1 0 —2 1 0 0 
asal az 4) 4=(9 9) Gan 
—1 1 0.8090 0.1910 0 0 
Ao = ( 2 -4 ) is ( 0.6179 0.3819 ) AS ( 1.236 0.764 ) nee) 
24 = (.764)*(0.1273, 0.0787) for k > 1 and xo = (0.255,0.0787). In equilibrium, P[L = 1] = 
(0.206)(0.764)! for 1 > 1 and P[L = 0] = 1/3. Thus, the distribution for L is a mixture of the 


geometric distribution and a distribution concentrated at zero. 
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1.14 Markov Fluid Model of a Queue 


A limiting form of quasi birth and death processes is a queue with a Markov-modulated rate of 
arrival, which is briefly described in this section. See D. Mitra, “Stochastic Theory of a Fluid 
Model of Producers and Consumers Coupled by a Buffer”, Annals of Applied Probability, Vol. 20, 
pp. 646-676, 1988 for more information. 

Let Q be the generator matrix of an irreducible Markov process with state space {1,..., m}. 
Suppose there is a buffer and that when the process is in state ¿, the net rate (arrival rate minus 
departure rate) of flow of fluid into the buffer is d;. The only exception to this rule is that if the 
buffer is empty and the Markov process is in a state i such that d; < 0 then the buffer simply 
remains empty. Let D denote the diagonal matrix with d1,...,dm along the diagonal. See Figure 
1.15. 


Markov process determines 


(input rate) - (output rate) 


Figure 1.15: Fluid queue with net input rate determined by a Markov process 


The average drift is defined to be >7;”, widi, or wDe, where w is the equilibrium probability 
distribution for the finite state Markov process and e is the column vector of all 1’s. 

Let L; denote the level of fluid in the buffer at time t and let 0, denote the state of the finite 
Markov process at time t. Then (L£¿,0¿) is a continuous state Markov process, and is a limiting 
form of a quasi birth-death process. In the remainder of this section we describe how to find the 
equilibrium distribution of this process. 

Let F(x,i,t) = P[0, = i, L¿ < x]. Kolmogorov’s forward equations are derived by beginning 
with the observation 


F(z,1,t+h) =hd F(z, j,t)qij + o(h) + F(a — hdi, i,t). (1.53) 
j=l 


Subtracting F'(x,7,t) from each side of this equation and letting h — 0 yields 


OF(z,i,t) € OF (a, 1, t) 
Ot = Pia Ox d (1 5 ) 


or in matrix notation, letting F(t) denote the row vector F(t) = (F(x,1,t),...,F(x,m,t)), 


Oe y (1.55) 
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Conditions on the equilibrium distribution F(z, i) are found by setting e = 0, yielding the equation 


FQ = “Ep. By definition, we know F(0o,1) = w;. Motivated by this and the theory of linear 
systems, we seek a solution of the form 


k 
F(a#,i) =w- y apie (1.56) 
i=1 


where (@;, zi) are eigenvalue-eigenvector pairs, meaning that ¢;Q = zi¢;D. 
To avoid substantial technical complications, we make the assumption that Q is reversible, 
meaning that wiqi = wjqji for 1 < i,j < m. Then all the eigenvalues are real valued. Let 


k = |{i : di > O}|. It can be shown that there are k strictly negative eigenvalues (counting 
multiplicity) and those are the ones to be included in the sum in (1.56). Finally, the k coefficients 
@1,...,@, can be found by satisfying the k boundary conditions: 

F(0,1) =0 whenever d; > 0 (1.57) 


For example, suppose 


TE ES and ae i) (1.58) 


We compute that w = (3, 2) and that (F(x,1), F(a, 2)) 


2exp(—x) 
— 3 


). In particular, in 


equilibrium, P[L; > x] = 


1.15 Problems 


1.1. Poisson merger 

Summing counting processes corresponds to “merging” point processes. Show that the sum of K 
independent Poisson processes, having rates \1,..., Ax, respectively, is a Poisson process with rate 
Ai +...+AxK. (Hint: First formulate and prove a similar result for sums of random variables, and 
then think about what else is needed to get the result for Poisson processes. You can use any one 
of the equivalent definitions given by Proposition 1.5.1 in the notes. Don't forget to check required 
independence properties.) 


1.2. Poisson splitting 

Consider a stream of customers modeled by a Poisson process, and suppose each customer is one 
of K types. Let (p1,...,px) be a probability vector, and suppose that for each k, the kt” customer 
is type ¿ with probability p;. The types of the customers are mutually independent and also inde- 
pendent of the arrival times of the customers. Show that the stream of customers of a given type 1 
is again a Poisson stream, and that its rate is \p;. (Same hint as in the previous problem applies.) 
Show furthermore that the K substreams are mutually independent. 


1.3. Poisson method for coupon collector’s problem 

(a) Suppose a stream of coupons arrives according to a Poisson process (A(t) : t > 0) with rate 
A = 1, and suppose there are k types of coupons. (In network applications, the coupons could be 
pieces of a file to be distributed by some sort ot gossip algorithm.) The type of each coupon in the 
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stream is randomly drawn from the k types, each possibility having probability i and the types of 
different coupons are mutually independent. Let p(k,t) be the probability that at least one coupon 
of each type arrives by time t. (The letter “p” is used here because the number of coupons arriving 
by time t has the Poisson distribution). Express p(k,t) in terms of k and t. 
(b) Find limg_... p(k, klnk + kc) for an arbitrary constant c. That is, find limit of the proba- 
bility that the collection is complete at time t = klnk + kc. (Hint: If a, —> a as k — oo, then 
(1+ on yk > e".) 
(c) The rest of this problem shows that the limit found in part (b) also holds if the total number of 
coupons is deterministic, rather than Poisson distributed. One idea is that if t is large, then A(t) 
is not too far from its mean with high probability. Show, specifically, that 

E / 
limpo P[A(kink + ke) > kink + he] = 4 9 r T 
(d) Let d(k,n) denote the probability that the collection is complete after n coupon arrivals. (The 
letter “d” is used here because the number of coupons, n, is deterministic.) Show that for any k, t, 
and n fixed, d(k,n)P[A(t) > n] < p(k, t) < P[A(t) > n] + P[A(t) < nja(k, n). 
(e) Combine parts (c) and (d) to identify limp... d(k, kInk + kc). 


1.4. The sum of a random number of random variables 

Let (X; : i > 0) be a sequence of independent and identically distributed random variables. Let 
S = Xı +... + Xy, where N is a nonnegative random variable that is independent of the sequence. 
(a) Express E[S] and Var(S) in terms of the mean and variance of X; and N. (Hint: Use the 
fact E[SF] = E[E[S*|N]]). (b) Express the characteristic function @g in terms of the characteristic 
function Px of Xi and the z-transform B(z) of N. 


1.5. Mean hitting time for a simple Markov process 
Let (X(n) : n > 0) denote a discrete-time, time-homogeneous Markov chain with state space 
{0,1,2,3} and one-step transition probability matrix 


0 1 0 0 

l-a 0 a 0 

oF 0 05 0 0.5 
0 0 1 0 


for some constant a with 0 < a < 1. (a) Sketch the transition probability diagram for X and 
give the equilibrium probability vector. If the equilibrium vector is not unique, describe all the 
equilibrium probability vectors. 

(b) Compute E[min{n > 1: X(n) = 3}|X(0) = 0]. 


1.6. A two station pipeline in continuous time 

Consider a pipeline consisting of two single-buffer stages in series. Model the system as a continuous- 
time Markov process. Suppose new packets are offered to the first stage according to a rate A Poisson 
process. A new packet is accepted at stage one if the buffer in stage one is empty at the time of 
arrival. Otherwise the new packet is lost. If at a fixed time t there is a packet in stage one and 
no packet in stage two, then the packet is transfered during [t,t + h) to stage two with probability 
hu. + o(h). Similarly, if at time t the second stage has a packet, then the packet leaves the system 
during [t,t + h) with probability huz + o(h), independently of the state of stage one. Finally, the 
probability of two or more arrival, transfer, or departure events during [t,t + h) is o(h). (a) What 


38 


is an appropriate state-space for this model? (b) Sketch a transition rate diagram. (c) Write down 
the Q matrix. (d) Derive the throughput, assuming that A = 1 = u2 = 1. (e) Still assuming 
A = 1 = a = 1. Suppose the system starts with one packet in each stage. What is the expected 
time until both buffers are empty? 


1.7. Simple population growth models 

A certain population of living cells begins with a single cell. Each cell present at a time t splits 
into two cells during the interval [t,t + h] with probability Ah + o(h), independently of the other 
cells, for some constant A > 0. The number of cells at time t, N(t), can thus be modeled as a 
continuous-time Markov chain. (a) Sketch an appropriate transition rate diagram and describe the 
transition rate matrix. (b) For fixed t > 0, let P(z,t) denote the z-transform of the probability 
vector T(t). Starting with the Kolmogorov forward equations for (1(t)), derive a partial differential 
equation for the z-transform of z(t). Show that the solution is 


—At 
ze 

P(z,t) = ———__,|, 

(2,1) l1—z+ze* 
(c) Find E[N(t)]. (d) Solve for m(t). (e) Another population evolves deterministically. The first cell 
splits at time A~!, and thereafter each cell splits into two exactly \~! time units after its creation. 
Find the size of the population of this population as a function of time, and compare to your answer 
in part (c). 


1.8. Equilibrium distribution of the jump chain 

Suppose that m is the equilibrium distribution for a time-homogeneous Markov process with tran- 
sition rate matrix Q. Suppose that B7! = Y; — qumi, Where the sum is over all ¿ in the state space, 
is finite. Show that the equilibrium distribution for the jump chain (X/(k) : k > 0) is given by 
n? = —Bquim;. (So m and Tr? are identical if and only if q; is the same for all i.) 


1.9. A simple Poisson process calculation 
1. Let (V(t) : t > 0) be a Poisson random process with rate A > 0. Compute P[N(s) = i|N(t) = k] 
where 0 < s < t and i and k are nonnegative integers. (Caution: note order of s and t carefully). 


1.10. An alternating renewal process 

Consider a two-color traffic signal on a street corner which alternately stays red for one minute and 
then stays green for one minute. Suppose you are riding your bicycle towards the signal and are 
a block away when you first spot it, so that it will take you one-half minute to reach the signal. 
(a) What is the probability that the signal will change colors (at least once) before you reach the 
corner? Explain. (b) Compute the conditional expectation of how long you must wait at the corner 
(possibly not at all) given that the light is green when you first spot it. Repeat assuming the light 
is red when you first spot it. (Which is greater? What is the average of the two answers?) (c) 
Repeat part (a) if instead the signal randomly changes colors, staying the same color for a time 
period uniformly distributed between 0 and 2 minutes. The times between switches are assumed 
to be independent. (d) How fast are you riding? (Hint: What units of speed can you use so that 
you don’t need more information?) 


1.10.5. A simple question of periods 
3. (5 points ) Consider a discrete-time Markov process with the nonzero one-step transition prob- 
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abilities indicated by the following graph. 


‘ae es (2 + (3) 
NG E NF 
7 X 
| } 
AS ~ AA 
SY SPN 

f | 
| | 
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CP IS ESO) 
SS NX) 


(a) What is the period of state 4? 
(b) What is the period of state 6? 


1.11. A mean hitting time problem 
Let (X(t) : t > 0) be a time-homogeneous, pure-jump Markov process with state space (0, 1,2) 
and Q matrix 


SAO: 2 
Q=| 1 -2 1 
2 0 -2 


(a) Write down the state transition diagram and compute the equilibrium distribution. 

(b) Compute a; = E[min{t > 0 : X(t) = 1}|X(0) = ¿] for i = 0,1,2. If possible, use an approach 
that can be applied to larger state spaces. 

(c) Derive a variation of the Kolmogorov forward differential equations for the quantities: a;(t) = 
PIX(s) # 2 for 0 < s < t and X(t) = i|X(0) = 0] for 0 < i < 2. (You need not solve the 
equations.) 

(d) The forward Kolmogorov equations describe the evolution of an initial probability distribu- 
tion going forward in time, given an initial. In other problems, a boundary condition is given 
at a final time, and a differential equation working backwards in time from a final condition is 
called for (called Kolmogorov backward equations). Derive a backward differential equation for: 
Bilt) = P[X(s) 42 fort < s < t| X(t) = j], for 0 < j < 2 and t < tf for some fixed time tp. (Hint: 
Express 3;(t—h) in terms of the P,(t)'s for t < tf, and let h — 0. You need not solve the equations.) 


1.12. A birth-death process with periodic rates 

Consider a single server queueing system in which the number in the system is modeled as a 
continuous time birth-death process with the transition rate diagram shown, where Aa, Ab, Ha, and 
y are strictly positive constants. 


ha Ah ha e ha 
R Te Te 3 Ta, 3 Poe 

ct ca ae 

ua lo, ua Pa ua 


(a) Under what additional assumptions on these four parameters is the process positive recurrent? 
(b) Assuming the system is positive recurrent, under what conditions on Aa, Ap, Ha, and up is it true 
that the distribution of the number in the system at the time of a typical arrival is the same as the 
equilibrium distribution of the number in the system? 
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1.13. Markov model for a link with resets 

Suppose that a regulated communication link resets at a sequence of times forming a Poisson 
process with rate u. Packets are offered to the link according to a Poisson process with rate A. 
Suppose the link shuts down after three packets pass in the absence of resets. Once the link is 
shut down, additional offered packets are dropped, until the link is reset again, at which time the 
process begins anew. 


(a) Sketch a transition rate diagram for a finite state Markov process describing the system state. 
(b) Express the dropping probability (same as the long term fraction of packets dropped) in terms 
of A and y. 


1.14. An unusual birth-death process 

Consider the birth-death process X with arrival rates A. = (p/(1 — p))*/az and death rates 
uk = (p/(1 — p))**/aj, where .5 < p < 1, and a = (ag,a7,...) is a probability distribution 
on the nonnegative integers with az > 0 for all k. (a) Classify the states for the process X as tran- 
sient, null-recurrent or positive recurrent. (b) Check that aQ = 0. Is a an equilibrium distribution 
for X? Explain. (c) Find the one-step transition probabilities for the jump-chain, X” (d) Classify 
the states for the process X7 as transient, null-recurrent or positive recurrent. 


1.15. A queue with decreasing service rate 

Consider a queueing system in which the arrival process is a Poisson process with rate A. Suppose 
the instantaneous completion rate is 4 when there are K or fewer customers in the system, and 1/2 
when there are K +1 or more customers in the system. The number in the system is modeled as a 
birth-death Markov process. (a) Sketch the transition rate diagram. (b) Under what condition on 
A and y are all states positive recurrent? Under this condition, give the equilibrium distribution. 
(c) Suppose that A = (2/3)u. Describe in words the typical behavior of the system, given that it is 
initially empty. 


1.16. Limit of a distrete time queueing system 

Model a queue by a discrete-time Markov chain by recording the queue state after intervals of q 
seconds each. Assume the queue evolves during one of the atomic intervals as follows: There is an 
arrival during the interval with probability aq, and no arrival otherwise. If there is a customer in 
the queue at the beginning of the interval then a single departure will occur during the interval 
with probability Pq. Otherwise no departure occurs. Suppose that it is impossible to have an 
arrival and a departure in a single atomic interval. (a) Find a¿=Plan interarrival time is kg] and 
b,=Pla service time is kq]. (b) Find the equilibrium distribution, p = (px : k > 0), of the number 
of customers in the system at the end of an atomic interval. What happens as q — 0? 


1.17. An M/M/1 queue with impatient customers 

Consider an M/M/1 queue with parameters A and y with the following modification. Each cus- 
tomer in the queue will defect (i.e. depart without service) with probability ah+o(h) in an interval 
of length h, independently of the other customers in the queue. Once a customer makes it to 
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the server it no longer has a chance to defect and simply waits until its service is completed and 
then departs from the system. Let N(t) denote the number of customers in the system (queue plus 
server) at time t. (a) Give the transition rate diagram and generator matrix Q for the Markov chain 
N =(N(t) :t> 0). (b) Under what conditions are all states positive recurrent? Under this condi- 
tion, find the equilibrium distribution for N. (You need not explicitly sum the series.) (c) Suppose 
that a = u. Find an explicit expression for pp, the probability that a typical arriving customer 
defects instead of being served. Does your answer make sense as A/¡u converges to zero or to infinity? 


1.18. Statistical multiplexing 

Consider the following scenario regarding a one-way link in a store-and-forward packet commu- 
nication network. Suppose that the link supports eight connections, each generating traffic at 5 
kilobits per second (kbps). The data for each connection is assumed to be in packets exponentially 
distributed in length with mean packet size 1 kilobit. The packet lengths are assumed mutually 
independent and the packets for each stream arrive according to a Poisson process. Packets are 
queued at the beginning of the link if necessary, and queue space is unlimited. Compute the mean 
delay (queueing plus transmission time—neglect propagation delay) for each of the following three 
scenarios. Compare your answers. (a) (Full multiplexing) The link transmit speed is 50 kbps. (b) 
The link is replaced by two 25 kbps links, and each of the two links carries four sessions. (Of course 
the delay would be larger if the sessions were not evenly divided.) (c) (Multiplexing over two links) 
The link is replaced by two 25 kbps links. Each packet is transmitted on one link or the other, and 
neither link is idle whenever a packet from any session is waiting. 


1.19. A queue with blocking 

(M/M/1/5 system) Consider an M/M/1 queue with service rate u, arrival rate A, and the modifi- 
cation that at any time, at most five customers can be in the system (including the one in service, 
if any). If a customer arrives and the system is full (i.e. already has five customers in it) then 
the customer is dropped, and is said to be blocked. Let N(t) denote the number of customers in 
the system at time t. Then (N(t) : t > 0) is a Markov chain. (a) Indicate the transition rate 
diagram of the chain and find the equilibrium probability distribution. (b) What is the proba- 
bility, pg, that a typical customer is blocked? (c) What is the mean waiting time in queue, W, 
of a typical customer that is not blocked? (d) Give a simple method to numerically calculate, 
or give a simple expression for, the mean length of a busy period of the system. (A busy period 
begins with the arrival of a customer to an empty system and ends when the system is again empty.) 


1.20. Multiplexing circuit and packet data streams 

Consider a communication link that is shared between circuit switched traffic (consisting of “calls” ) 
and datagram packet traffic. Suppose new calls arrive at rate Ac, that new packets arrive at rate Ap, 
and that the call durations are exponential with parameter ôe. All calls are “served” simultaneously. 
Suppose that the link capacity is exactly sufficient to carry C calls, and that calls have priority 
over packets. Thus, a new call is blocked and lost if and only if it arrives to find C calls already in 
progress. Packets are not blocked, but instead they are queued until transmission. Finally, suppose 
that the instantaneous service rate of packets is u(C — ne) when there are ne calls in progress. 
(a) Define a continuous-time, countable state Markov chain to model the system, and indicate the 
transition rates. (b) Is the method of Markov processes with phases relevant to this problem? If 
so, describe in detail how the method applies. (c) What is the necessary and sufficient condition 
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for the system to be stable (be as explicit as possible). 


1.20.1. Three queues and an autonomously traveling server 
Consider three stations that are served by a single rotating server, as pictured. 


——== station 1 == 


— station 2 


pio? 


3 i Yy 
—>| station 3 == 


Customers arrive to station į according to a Poisson process of rate A; for 1 < i < 3, and the total 
service requirement of each customer is exponentially distributed, with mean one. The rotation 
of the server is modelled by a three state Markov process with the transition rates a, 3, and y as 
indicated by the dashed lines. When at a station, the server works at unit rate, or is idle if the 
station is empty. If the service to a customer is interrupted because the server moves to the next 
station, the service is resumed when the server returns. 

(a) Under what condition is the system stable? Briefly justify your answer. 

(b) Identify a method for computing the mean customer waiting time at station one. 


1.21. On two distibutions seen by customers 

Consider a queueing system in which the number in the system only changes in steps of plus one 
or minus one. Let D(k,t) denote the number of customers that depart in the interval [0,t] that 
leave behind exactly k customers, and let R(k,t) denote the number of customers that arrive in the 
interval [0,t] to find exactly k customers already in the system. (a) Show that |D(k,t)— R(k,t)| < 1 
for all k and t. (b) Let a; (respectively 6; ) denote the number of arrivals (departures) up to time 
t. Suppose that a; — co and a;/d; — 1 as t > oo. Show that if the following two limits exist for 
a given value k, then they are equal: rg = limy.. R(k, t)/az and dk = limy. D(k,t)/0;. 
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Chapter 2 


Foster-Lyapunov stability criterion 
and moment bounds 


Communication network models can become quite complex, especially when dynamic scheduling, 
congestion, and physical layer effects such as fading wireless channel models are included. It is thus 
useful to have methods to give approximations or bounds on key performance parameters. The 
criteria for stability and related moment bounds discussed in this chapter are useful for providing 
such bounds. 

Aleksandr Mikhailovich Lyapunov (1857-1918) contributed significantly to the theory of stabil- 
ity of dynamical systems. Although a dynamical system may evolve on a complicated, multiple 
dimensional state space, a recurring theme of dynamical systems theory is that stability questions 
can often be settled by studying the potential of a system for some nonnegative potential function 
V. Potential functions used for stability analysis are widely called Lyapunov functions. Similar 
stability conditions have been developed by many authors for stochastic systems. Below we present 
the well known criteria due to Foster [14] for recurrence and positive recurrence. In addition we 
present associated bounds on the moments, which are expectations of some functions on the state 
space, computed with respect to the equilibrium probability distribution.! 

Section 2.1 discusses the discrete time tools, and presents examples involving load balancing 
routing, and input queued crossbar switches. Section 2.2 presents the continuous time tools, and 
an example. Problems are given in Section 2.3. 


2.1 Stability criteria for discrete time processes 


Consider an irreducible discrete-time Markov process X on a countable state space S, with one-step 
transition probability matrix P. If f is a function on S, then Pf represents the function obtained 
by multiplication of the vector f by the matrix P: Pf (i) = yes pijf (J). If f is nonnegative, then 


The proof of Foster's criteria given here is similar to Foster's original proof, but is geared to establishing the 
moment bounds, and the continuous time versions, at the same time. The moment bounds and proofs given here 
are adaptations of those in Meyn and Tweedie [30] to discrete state spaces, and to continuous time. They involve 
some basic notions from martingale theory, which can be found, for example, in [18]. A special case of the moment 
bounds was given by Tweedie [42], and a version of the moment bound method was used by Kingman [26] in a 
queueing context. As noted in [30], the moment bound method is closely related to Dynkin’s formula. The works 
[39, 40, 28, 41], and many others, have demonstrated the wide applicability of the stability methods in various 
queueing network contexts, using quadratic Lyapunov functions. 
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Pf is well defined, with the understanding that Pf(i) = +00 is possible for some, or all, values of 
i. An important property of Pf is that Pf(i) = E[f(X(t+ 1)|X(t) = 1]. Let V be a nonnegative 
function on S, to serve as the Lyapunov function. The drift function of V(X(t)) is defined by 
dli) = E[V(X(t+1))|X(t) = iJ- V (i). That is, d= PV —V. Note that d(i) is always well-defined, 
if the value +00 is permitted. The drift function is also given by 


d(i) = Y pi(V(5) — V (i). (2.1) 
jjżi 


Proposition 2.1.1 (Foster-Lyapunov stability criterion) Suppose V : S > Ry and C is a finite 
subset of S. 

(a) If {i: V(i) < K} is finite for all K, and if PV — V <0 on S — C, then X is recurrent. 

(b) Ife >0 and b is a constant such that PV — V < —e + bIc, then X is positive recurrent. 


The proof of the proposition is given after two lemmas. 


Lemma 2.1.2 Suppose PV — V < -f +g on S, where f and g are nonnegative functions. Then 
for any initial state io and any stopping time T, 


E| Y) FX) |<VG)J+E| X, Xk) (2.2) 


k:0<k<r—1 k:0<k<r—1 


Proof. Let Fẹ denote the o-algebra generated by (X(s) : 0< s < k). By assumption, PV + f < 
V +g on S. Evaluating each side of this inequality at X(k), and taking the conditional expectation 
given Fy yields 

ELV (x(k + 1))|Fe] + HX(k)) < V(X(k)) + g(X(4)). (2.3) 


Let 7” = min{7,n,inf{k > 0: V(X(k)) > nj). The next step is to multiply each side of (2.3) by 
Tizn>ky, take expectations of each side, and use the fact that 


E [E[V(<(k + DIA] ros] = E [V(e(k + 1))Lensey] > E [V(e(k +1) Lóm>+1)) - 
The result is 
E[V(a(k +1) Lira] + E [F (2(k) irns] < E [V(x(k) insg] + E [gle(k) en>] (2-4) 


The definition of 7” implies that all the terms in (2.4) are zero for k > n, and that E [V(x(k))L(n>13] < 
oo for all k. Thus, it is legitimate to sum each side of (2.4) and cancel like terms, to yield: 


E SS AX(M)|<VG)+E| Y g(X(%)|. (2.5) 
k:0<k<rr—1 k:0<k<rr—1 
Letting n — oo in (2.5) and appealing to the monotone convergence theorem yields (2.2). pa 


Lemma 2.1.3 Let X be an irreducible, time-homogeneous Markov process. If there is a finite set 
C, and the mean time to hit C starting from any state in C is finite, then X is positive recurrent. 
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Proof. Lemma 1.3.9 of Assussen. Variation of Wald's lemma. E 


Proof of Proposition 2.1.1. (a) The proof of part (a) is based on a martingale convergence 
theorem. Fix an initial state i, € S — C, let 7 = min{t > 0 : X(t) € C}, and let Y(t) = V(X(tA7)) 
for t > 0. Note that Y(t +1) — Y (t) = (V(X(t +1) — V(X(t)))L(r>+)- Let F denote the o-algebra 
generated by (X(s):0 < s < t), and note that the event (7 > t} is in F;. Thus, 


ElY (t+1)—-Y (t)| F] = ELV (X0) -V X Hry Fi] = EV XE -V(X F>} 2 0, 


so that (Y, : t > 0) is a nonnegative supermartingale. Therefore, by a version of the martingale 
convergence theorem, lim... Y (t) exists and is finite with probability one. By the assumption 
Li: V(i) < K} is finite for all K, it follows that X(t A 7) reaches only finitely many states with 
probability one, which implies that r < +00 with probability one. Therefore, P[r < œ|X(0) = 
io] = 1 for any i, € S — C. Therefore, for any initial state in C, the process returns to C infinitely 
often with probability one. The process watched in C is a finite-state Markov process, and is thus 
(positive) recurrent. Therefore, all the states of C are recurrent for the original process X. Part 
(a) is proved. 

(b) Let f = -e, g = blo, and T = min{t > 1: X(t) € C}. Then Lemma 2.1.2 implies that 
eEl[r] < V(io) +b for any io € S. In particular, the mean time to hit C after time zero, beginning 
from any state in C, is finite. Therefore X is positive recurrent by Lemma 2.1.3. E 


Proposition 2.1.4 (Moment bound) Suppose V, f, and g are nonnegative functions on S and 


suppose 
PV(i) —V(i) < -f +gli)  forallies (2.6) 


In addition, suppose X is positive recurrent, so that the means, f = nf and J = rg are well-defined. 
Then f <J. (In particular, if g is bounded, then y is finite, and therefore f is finite.) 


Proof. Fix a state i, and let Tm be the time of the m*” return to state io. Then by the equality 
of time and statistical averages, 


E >X f(X(A))} = mEn] 


{k:0<k<Tm—1} 


E >X OIXK)| = mE(TIG 
{k:0<k<Tm—1} 


Lemma 2.1.2 applied with stopping time Tm thus yields mE[T;] f < V (io) + mE|T,|g. Dividing 
through by mE|T}] and letting m — oo yields the desired inequality, f < g. a 


Corollary 2.1.5 (Combined Foster-Lyapunov stability criterion and moment bound) Suppose V, f, 
and g are nonnegative functions on S such that 


PV(i) -—V(i) <—f()+g(i)  foralies (2.7) 


In addition, suppose for some € > 0 that the set C defined by C = {i: f(i) < g(t) +e} is finite. Then 
X is positive recurrent and f < 3. (In particular, if g is bounded, then y is finite, and therefore f 
is finite. ) 


AT 


Je queue 1 {> 

— y i 
l d 
Sl queue 2 pa 


Figure 2.1: Two queues fed by a single arrival stream. 


Proof. Let b = max{g(i) +e— f(i) : i € C}. Then V,C,b, and e satisfy the hypotheses of 
Proposition 2.1.1(b), so that X is positive recurrent. Therefore the hypotheses of Proposition 2.1.4 
are satisfied, so that f < 7. E 


The assumptions in Propositions 2.1.1 and 2.1.4 and Corollary 2.1.5 do not imply that V is 
finite. Even so, since V is nonnegative, for a given initial state X (0), the long term average drift of 
V(X(t)) is nonnegative. This gives an intuitive reason why the mean downward part of the drift, 
f, must be less than or equal to the mean upward part of the drift, g. 


Example la (Probabilistic routing to two queues) Consider the routing scenario with two 
queues, queue 1 and queue 2, fed by a single stream of packets, as pictured in Figure 2.1. Here, 
0 < a,u, dı,d2 < 1, and u = 1 — u. The state space for the process is S = ZÊ, where the state 
x = (11,12) denotes xı packets in queue 1 and x2 packets in queue 2. In each time slot, a new 
arrival is generated with probability a, and then is routed to queue 1 with probability u and to 
queue 2 with probability u. Then each queue 2, if not empty, has a departure with probability di. 
Note that we allow a packet to arrive and depart in the same slot. Thus, if X;(t) is the number of 
packets in queue 1 at the beginning of slot t, then the system dynamics can be described as follows: 


Xi(t+1) = X;(t) + As(t) — Di(t) + Lilt) for i € {0,1} (2.8) 
where 
e A(t) = (Aı (t), Aa(t)) is equal to (1,0) with probability au, (0,1) with probability au, and 
A(t) = (0,0) otherwise. 
e D,(t):t > 0, are Bernoulli(d;) random variables, for i € {0,1} 
e All the A(t)’s, D¡(t)'s, and Da(t)'s are mutually independent 
e L(t) = (—(X;(t) + As(t) — D;(t)))+ (see explanation next) 


If X;(t) + A;(t) = 0, there can be no actual departure from queue i. However, we still allow D;(t) 
to equal one. To keep the queue length process from going negative, we add the random variable 
L;(t) in (2.8). Thus, D;(t) is the potential number of departures from queue 7 during the slot, and 
D,(t) — L;(t) is the actual number of departures. This completes the specification of the one-step 
transition probabilities of the Markov process. 

A necessary condition for positive recurrence is, for any routing policy, a < dı + d2, because the 
total arrival rate must be less than the total depature rate. We seek to show that this necessary 
condition is also sufficient, under the random routing policy. 
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Let us calculate the drift of V(X(t)) for the choice V(x) = (a7 +23) /2. Note that (X;(t+1))? = 
(X;(t) + As(t) — Dile) + L;(t))? < (Xi(t) + Ai(t) — Di(t))?, because addition of the variable L;(t) 
can only push X;(t) + A;(t) — D;(t) closer to zero. Thus, 

PV(x)—- V(x) = E[V(X(E4+1))|X@® = 2] — V(x) 


Ele; + Ai(t) — Di(t))? — 271X(t) = 2] 


IA 
NI =e 


i=1 
2 
= S ci E[As(t) — D,(t)|X(t) = z] + SBI Ait — D,(t))?|X(t) = x] (2.9) 
i=l 
2 
< (> r¡E[A,(t) — Di(t)|X(t) = J +1 
¡=1 
= —(21(d; — au) + z2(d2 — au)) + 1 (2.10) 


Under the necessary condition a < dı + d2, there are choices of u so that au < dı and at < dg, and 
for such u the conditions of Corollary 2.1.5 are satisfied, with f(x) = zı(dı — au) + xa(da — au), 
g(x) = 1, and any e > 0, implying that the Markov process is positive recurrent. In addition, the 
first moments under the equlibrium distribution satisfy: 


(dı — au)X 1 + (da — at) X2 < 1. (2.11) 


In order to deduce an upper bound on Xj + Xo, we select u* to maximize the minimum of the 
two coefficients in (2.11). Intuitively, this entails selecting u to minimize the absolute value of the 
difference between the two coefficients. We find: 


e = max min{d) — au, dz — au} 
0<u<1 
dy + da — 
= miníd;, da, ata) 


and the corresponding value u* of u is given by 


y if dj — dọ < —a 
u* = L ¿4 dde if ld; =da <a 
i if dy = da >a 
For the system with u = u*, (2.11) yields 
a 1 
X1i+X29<-. (2.12) 
€ 
We remark that, in fact, 
2 
Xı+X E 2.13 
¡FAS EE ( ) 


If [dí — d2| < a then the bounds (2.12) and (2.13) coincide, and otherwise, the bound (2.13) is 
strictly tighter. If dı — da < —a then u* = 0, so that X; = 0, and (2.11) becomes (dz — a)X2 < 1 
, which implies (2.13). Similarly, if dı — da > a, then u* = 1, so that X2 = 0, and (2.11) becomes 
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(dı — a)Xı < 1, which implies (2.13). Thus, (2.13) is proved. 


Example 1b (Route-to-shorter policy) Consider a variation of the previous example such that 
when a packet arrives, 1t is routed to the shorter queue. To be definite, in case of a tie, the packet 
is routed to queue 1. Then the evolution equation (2.8) still holds, but with with the description 
of the arrival variables changed to the following: 


e Given X(t) = (21,12), A(t) = (Lío,<oy L(01>0,) with probability a, and A(t) = (0,0) 
otherwise. Let PS denote the one-step transition probability matrix when the route-to- 
shorter policy is used. 


Proceeding as in (2.9) yields: 


PRSY (a) — V(x) 


IA 


2 
S > aiB[A,(t) — Di(t))|X(t) = 2] +1 
4=1 


= a (21L(0,<a07) + 22L(0,>07)) — 4121 — daa +1 


Note that 21L(7,<n,) + 12L(0,>0,) < UTI + Ux2 for any u € [0,1], with equality for u = Itri <z}: 
Therefore, the drift bound for V under the route-to-shorter policy is less than or equal to the drift 
bound (2.10), for V for any choice of probabilistic splitting. In fact, route-to-shorter routing can 
be viewed as a controlled version of the independent splitting model, for which the control policy is 
selected to minimize the bound on the drift of V in each state. It follows that the route-to-shorter 
process is positive recurrent as long as a < dı + da, and (2.11) holds for any value of u such that 
au < dı and au < dg. In particular, (2.12) holds for the route-to-shorter process. 

We remark that the stronger bound (2.13) is not always true for the route-to-shorter policy. 
The problem is that even if dí — da < —a, the route-to-shorter policy can still route to queue 1, 
and so Xı 4 0. In fact, if a and da are fixed with 0 < a < da < 1, then X; > œ as dı — 0 for 
the route-to-shorter policy. Intuitively, that is because occasionally there will be a large number of 
customers in the system due to statistical fluctuations, and then there will be many customers in 
queue 1. But if da << 1, those customers will remain in queue 2 for a very long time. 

Example 2a (An input queued switch with probabilistic switching) ? Consider a packet switch 
with N inputs and N outputs, as pictured in Figure 2.2. Suppose there are N? queues — N at 
each input — with queue 2, j containing packets that arrived at input 7 and are destined for output 
j, for i,j € E, where E = [1,--- ,N}. Suppose the packets are all the same length, and adopt 
a discrete time model, so that during one time slot, a transfer of packets can occur, such that at 
most one packet can be transferred from each input, and at most one packet can be transferred to 
each output. A permutation o of E has the form o = (04,...,0n), where o1,...,0y are distinct 
elements of E. Let II denote the set of all N! such permutations. Given o € II, let R(o) be the 
N x N switching matriz defined by Rij = Iy,,-;;. Thus, Rij (o) = 1 means that under permutation 
o, input 7 is connected to output j, or, equivalently, a packet in queue 7,7 is to depart, if there is 
any such packet. A state x of the system has the form x = (aj; : i,j € E), where zij denotes the 
number of packets in queue 2, j. 


? Tassiulas [41] originally developed the results of Examples 2a-b, in the context of wireless networks. The paper 
[29] presents similiar results in the context of a packet switch. 
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Figure 2.2: A 4 x 4 input queued switch 


The evolution of the system over a time slot [t,t + 1) is described as follows: 
Xij(t +1) = Xiglt) + Aig) — Rigo) + Lis(t) 
where 


e Aj;(t) is the number of packets arriving at input i, destined for output j, in the slot. Assume 
that the variables (4;5(t) : i,j € E,t > 0) are mutually independent, and for each i,j, the 
random variables (A;;(t) : t > 0) are independent, identically distributed, with mean A;; and 
E[A;,] < Kij, for some constants Aj; and Kij. Let A = (Ai; : i,j € E). 


e a(t) is the switch state used during the slot 


o Lij = (—(Xi¡(t) + Aij(t) — Ris(0(t)))+, which takes value one if there was an unused potential 
departure at queue ij during the slot, and is zero otherwise. 


The number of packets at input 2 at the beginning of the slot is given by the row sum 
jee Xij(t), its mean is given by the row sum > ¡eg Aij, and at most one packet at input i 
can be served in a time slot. Similarly, the set of packets waiting for output j, called the virtual 
queue for output j, has size given by the column sum > ¡¿g Xij(t). The mean number of arrivals 
to the virtual queue for output j is > ¿eg Aij(t), and at most one packet in the virtual queue can 
be served in a time slot. These considerations lead us to impose the following restrictions on A: 


y Aij < 1 for alli and S Ag < 1 for all j (2.14) 
jEE 1€E 


Except for trivial cases involving deterministic arrival sequences, the conditions (2.14) are necessary 
for stable operation, for any choice of the switch schedule (a(t) : t > 0). 

Let’s first explore random, independent and identically distributed (i.i.d.) switching. That is, 
given a probability distribution u on II, let (a(t) : t > 0) be independent with common probability 
distribution u. Once the distributions of the A;;’s and u are fixed, we have a discrete-time Markov 
process model. Given A satisfying (2.14), we wish to determine a choice of u so that the process 
with i.i.d. switch selection is positive recurrent. 

Some standard background from switching theory is given in this paragraph. A line sum of a 
matrix M is either a row sum, )> j Mij, or a column sum, >>, Mij. A square matrix M is called 
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doubly stochastic if it has nonnegative entries and if all of its line sums are one. Birkhoff’s theorem, 
celebrated in the theory of switching, states that any doubly stochastic matrix M is a convex 
combination of switching matrices. That is, such an M can be represented as M => ¿y R(0)u(o), 
where u = (u(o) : o € I) is a probability distribution on II. If M is a nonnegative matrix with all 
line sums less than or equal to one, then if some of the entries of M are increased appropriately, 
a doubly stochastic matrix can be obtained. That is, there exists a doubly stochastic matrix M 
so that Mj; < Mi; for all i, j. Applying Birkhoff’s theorem to M yields that there is a probability 
distribution u so that Mi; < <> sen R(o)u(c) for all i, j. 

Suppose A satisfies the necessary conditions (2.14). That is, suppose that all the line sums of 
A are less than one. Then with e defined by 

pe 1 — (maximum line sum of A) 


N 


each line sum of (Ajj + € : i,j € E) is less than or equal to one. Thus, by the observation at the 
end of the previous paragraph, there is a probability distribution u* on II so that Ay; +€ < pi; (u*), 


where 
mlu) = > Risto 
oell 


We consider the system using probability distribution u* for the switch states. That is, let (a(t) : 
t > 0) be independent, each with distribution u*. Then for each ij, the random variables R;¿(o(t)) 
are independent, Bernoulli(j;;(u*)) random variables. 

Consider the quadratic Lyapunov function V given by V(x) = 3), 3 xj, . As in (2.9), 


PV(x) =V a) < Y ropan O — Ry(o)|Xy = 2] + ee Rilo) IXE) = x] 
ij i,j 
Now 
E| Ay (t) — Ryl) | Xy) = 2] = E[A¡;(t) — Rislo(t))] = Aiz — mizlu*) < —€ 
and 


where K = 4(N + Žij Kij). Thus, 
PV(x)— V(x) < —e Sy +K (2.15) 


Therefore, by Corollary 2.1.5, the process is positive recurrent, and 
D < (2.16) 


That is, the necessary condition (2.14) is also sufficient for positive recurrence and finite mean queue 
length in equilibrium, under i.i.d. random switching, for an appropriate probability distribution u* 
on the set of permutations. 
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Example 2b (An input queued switch with maximum weight switching) The random switching 
policy used in Example 2a depends on the arrival rate matrix A, which may be unknown a priori. 
Also, the policy allocates potential departures to a given queue ij, whether or not the queue is 
empty, even if other queues could be served instead. This suggests using a dynamic switching 
policy, such as the maximum weight switching policy, defined by a(t) = o@“ (X(t)), where for a 
state zx, 


MW 
= ij Ry; (Co). 2.1 
gre) are max 2 Dipl ) (2.17) 


The use of “arg max” here means that oW (x) is selected to be a value of ø that maximizes the 
sum on the right hand side of (2.17), which is the weight of permutation o with edge weights x;;. In 
order to obtain a particular Markov model, we assume that the set of permutations IT is numbered 
from 1 to N! in some fashion, and in case there is a tie between two or more permutations for having 
the maximum weight, the lowest numbered permutation is used. Let PMW denote the one-step 
transition probability matrix when the route-to-shorter policy is used. 


Letting V and K be as in Example 2a, we find under the maximum weight policy that 


PMWYV (2) — V(x) < O rs Foe (x))) +K 
ij 


The maximum of a function is greater than or equal to the average of the function, so that for any 
probability distribution u on H 


y xi Rij (go (t)) 
tj 


IV 


Y ulo) S agaaa) (2.18) 
o ij 
= Y egw) 
aj 


with equality in (2.18) if and only if u is concentrated on the set of maximum weight permutations. 
In particular, the choice u = u* shows that 


NN ta @) => ales Os +O 


Therefore, if P is replaced by PMW, (2.15) still holds. Therefore, by Corollary 2.1.5, the process 
is positive recurrent, and the same moment bound, (2.16), holds, as for the randomized switching 
strategy of Example 2a. On one hand, implementing the maximum weight algorithm does not 
require knowledge of the arrival rates, but on the other hand, it requires that queue length infor- 
mation be shared, and that a maximization problem be solved for each time slot. Much recent 
work has gone towards reduced complexity dynamic switching algorithms. 


2.2 Stability criteria for continuous time processes 
Here is a continuous time version of the Foster-Lyapunov stability criteria and the moment bounds. 
Suppose X is a time-homegeneous, irreducible, continuous-time Markov process with generator 


matrix Q. The drift vector of V(X(t)) is the vector QV. This definition is motivated by the fact 
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that the mean drift of X for an interval of duration h is given by 


E|V(X(t+ h))| X(t) = 1] - V(s) 


dy(i) i 
= pij(h) — dij VG) 
2 ar dá 
= E (u+) vo (2.19) 
JES 


so that if the limit as h — 0 can be taken inside the summation in (2.19), then d, (1) — QV (i) as 
h — 0. The following useful expression for QV follows from the fact that the row sums of Q are 
Zero: 
QV(i) = Y gV G) - V(). (2.20) 
FIA 


Formula (2.20) is quite similar to the formula (2.1) for the drift vector for a discrete-time process. 


Proposition 2.2.1 (Foster-Lyapunov stability criterion—continuous time) Suppose V : S > Ry 
and C is a finite subset of S. 

(a) If QV <0 on S—C, and {i: V(t) < K} is finite for all K then X is recurrent. 

(b) Suppose for some b > 0 and e > 0 that 


QV (i) < —e + blo(i) for allie S. (2.21) 


Suppose further that {i : V(i) < K} is finite for all K, or that X is nonexplosive. Then X is 
positive recurrent. 


The proof of the proposition is similar to that above for discrete time processes. We begin with 
an analog of Lemma 2.1.2. 


Lemma 2.2.2 Suppose QV < —f +g on S, where f and g are nonnegative functions. Fix an 
initial state io, let N be a stopping time for the jump process X7, and let TN denote the time of 
the N* jump of X. Then 


N 


E / FX(0)dt| < Vli) + E / a(X (t))dt (2.22) 


Proof. Let D denote the diagonal matrix with entries —q;;, The one-step transition probability 
matrix of the jump chain is given by P? = D1Q + I. The condition QV < —f + g thus implies 
PIV -V < -f +J, were f = D~'f and J = D~'g. Lemma 2.1.2 applied to the jump chain thus 
yields 


El Y f(X%(k)| <VG)+E} X aX (b) (2.23) 
k:0<k<r—1 k:0<k<r—1 
However, by the space-time description of an excursion of X from io, 


E © Fw f pocos 


k:0<k<N—1 
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and a similar equation holds for y and g. Substituting these relations into (2.23) yields (2.22), as 
desired. | 


Proof of Proposition 2.2.1. (a) Let D denote the diagonal matrix with entries —q;;. The 
one-step transition probability matrix of the jump chain is given by P? = D~'Q+TI. The condition 
QV < 0 thus implies PV — V < 0, so that X/ is recurrent by Proposition 2.1.1(a). But X7 is 
recurrent if and only if X is recurrent, so X is also recurrent. 

(b) The assumptions imply QV < 0 on S —C, so if {i : V(i) < K} is finite for all K then X is 
recurrent by Proposition 2.2.1(a), and, in particular, X is nonexplosive. Thus, X is nonexplosive, 
either by direct assumption or by implication. 

Let f = —e, let g = blo, let iy € C, let N = min{k > 1: X“(k) € C}, and let Ty denote 
the time of the N*” jump. Then Lemma 2.2.2 implies that eE[TWN] < V(io) + b/qi,i,. Since Ty is 
finite a.s. and since X is not explosive, it must be that Ty is the time that X returns to the set C 
after exiting the initial state io. That is, the time to hit C beginning from any state in C is finite. 
Therefore X is positive recurrent by the continuous time version of Lemma 2.1.3. E 


Example 3 Suppose X has state space S = Z4, with qio = pw for all i > 1, qui41 = As for 
all ¿ > 0, and all other off-diagonal entries of the rate matrix Q equal to zero, where y > 0 and 
Ai > 0 such that eine < +00. Let C = {0}, V(0) = 0, and V(t) = 1 for i > 0. Then 
QV = —u + (ào + lc, so that (2.21) is satisfied with e = u and b = Ap + u. However, X is not 
positive recurrent. In fact, X is explosive. To see this, note that py, u= ESY > exp(— £). Let 
ô be the probability that, starting from state 0, the jump process does not return to zero. Then 
ô = [R25 Pj > exp(—u izo x) > 0. Thus, X” is transient. After the last visit to state zero, all 
the jumps of X” are up one. The corresponding mean holding times of X are x which have a 
finite sum, so that the process X is explosive. This example illustrates the need for the assumption 
just after (2.21) in Proposition 2.2.1. 


As for the case of discrete time, the drift conditions imply moment bounds. The proofs of the 
following two propositions are minor variations of the ones used for discrete time, with Lemma 
2.1.2 used in place of Lemma 2.2.2, and are omitted. 


Proposition 2.2.3 (Moment bound-continuous time) Suppose V, f, and g are nonnegative func- 
tions on S, and suppose QV(i) < —f(t) + g(t) for alli € S. In addition, suppose X is positive 


recurrent, so that the means, f = rf and y = rg are well-defined. Then f < g. 


Corollary 2.2.4 (Combined Foster-Lyapunov stability criterion and moment bound-continuous 
time) Suppose V, f, and g are nonnegative functions on S such that QV(i) < —f(t) + gli) for all 
i € S, and, for some e > 0, the set C defined by C = {i : f(t) < g(t) + e) is finite. Suppose also 
that Li: V(i) < K} is finite for all K. Then X is positive recurrent and f < J. 


Example 4.a (Random server allocation with two servers) Consider the system shown in Figure 
2.3. Suppose that each queue 2 is fed by a Poisson arrival process with rate A;, and suppose there 
are two potential departure processes, Dı and D2, which are Poisson processes with rates mı and 
ma, respectively. The five Poisson processes are assumed to be independent. No matter how the 
potential departures are allocated to the permitted queues, the following conditions are necessary 
for stability: 

A < may, Az < ma, and A1 + à2 + A3 < m1 + mo (2.24) 
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Figure 2.3: A system of three queues with two servers 


That is because server 1 is the only one that can serve queue 1, server 2 is the only one that can 
serve queue 3, and the sum of the potential service rates must exceed the sum of the potential 
arrival rates for stability. A vector x = (£1, £2, £2) € Ze corresponds to x; packets in queue 2 for 
each 2. Let us consider random selection, so that when D; has a jump, the queue served is chosen at 
random, with the probabilities determined by u = (u1, u2). As indicated in Figure 2.3, a potential 
service by server 1 is given to queue 1 with probability u1, and to queue 2 with probability u;. 
Similarly, a potential service by server 2 is given to queue 2 with probability u2, and to queue 3 
with probability uz. The rates of potential service at the three stations are given by 


mlu) = um; 
pa(u) = UM] + ugme 
ps(u) = uma. 


Let V(x) = 5(2 + 23 + 23). Using (2.20), we find that the drift function QV is given by 
3 


2 


3 
ova) = (Ne +12] +2 [Y 04 —22ula) 
2 2 


1 


Now (a; — 1)4 < (a; — 1), so that 


3 
QV (z) < (> ri(Ai — mea) ite > (2.25) 
i=l 


where y is the total rate of events, given by y = A1 + A2 +A3 + u1 (u) + u2(u)+ u3(u), or equivalently, 
y =A1+A2+A3+m1 + mz. Suppose that the necessary condition (2.24) holds. Then there exists 
some € > 0 and choice of u so that 


Ai +e< mlu) fr 1<i<3 


and the largest such choice of e is e = min{m, — A1,ma (See excercise.) 
So QV (1) < —e(11 + 12 + 23) + y for all x, so Corollary 2.2.4 implies that X is positive recurrent 
and X1 +X2 + X3 < ae. 


m1+m9a—A1 —A2—A: 
A3, 1 2 Au 2 22. 
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Example 4.b (Longer first server allocation with two servers) This is a continuation of Example 
4.a, concerned with the system shown in Figure 2.3. Examine the right hand side of (2.25). Rather 
than taking a fixed value of u, suppose that the choice of u could be specified as a function of the 
state x. The maximum of a function is greater than or equal to the average of the function, so that 
for any probability distribution u, 


IA 


3 
y 5 [li (u) 
i=1 


max y a pi(u!) (2.26) 


= max mi(214, + equi) + ma(230) + 23u,) 
u 


= mi(211 V 12) + ma(x23 V 23) 


with equality in (2.26) for a given state x if and only if a longer first policy is used: each service 
opportunity is allocated to the longer queue connected to the server. Let Q4" denote the one-step 
transition probability matrix when the longest first policy is used. Then (2.25) continues to hold 
for any fixed u, when Q is replaced by Q/". Therefore if the necessary condition (2.24) holds, e 
can be taken as in Example 4a, and Q// V(x) < —e(x1 + 22 + 23) + y for all x. So Corollary 2.2.4 
implies that X is positive recurrent under the longer first policy, and X; + Xə + X; < ee (Note: 
We see that 


3 
QV (x) < (> za) — mi (11 V 23) — M2(a2 V £3) + 7 
i=1 


but for obtaining a bound on X;+ X2+-X3 it was simpler to compare to the case of random service 
allocation.) 


2.3 Problems 


2.1. Recurrence of mean zero random walks 

(a) Suppose B1, Bo,... is a sequence of independent, mean zero, integer valued random variables, 
which are bounded, i.e. P[|B;| < M] = 1 for some M. 

(a) Let Xy = 0 and Xn = Bı +---+ Bn for n > 0. Show that X is recurrent. 

(b) Suppose Yo = 0 and Y,41 = Yn + Bn + Ln, where Ln = (—(Yn + Bn))+. The process Y is a 
reflected version of X. Show that Y is recurrent. 


2.2. Positive recurrence of reflected random walk with negative drift 

Suppose B1, Bo,... is a sequence of independent, integer valued random variables, each with mean 
B < 0 and second moment B2 < +00. Suppose Xo = 0 and Xn+1 = Xn + Bn + Ln, where 
Ln = (—(Xn + Bn))+. Show that X is positive recurrent, and give an upper bound on the mean 


under the equilibrium distribution, X. (Note, it is not assumed that the B’s are bounded.) 


2.3. Routing with two arrival streams 
(a) Generalize Example 1.a to the scenario shown. 
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where as, d; € (0,1) for 1 <i < 2 and 1 < j < 3. In particular, determine conditions on a; and 
az that insure there is a choice of u = (u1, u2) which makes the system positive recurrent. Under 
those conditions, find an upper bound on X; + Xə + X3, and select u to mnimize the bound. 

(b) Generalize Example 1.b to the scenario shown. In particular, can you find a version of route- 
to-shorter routing so that the bound found in part (a) still holds? 


2.4. An inadequacy of a linear potential function 
Consider the system of Example 1.b in the notes (a discrete time model, using the route to shorter 
policy, with ties broken in favor of queue 1, so u = Itr; <z9}): 


queue 1 {> 
Ja” 


Assume a = 0.7 and dı = d2 = 0.4. The system is positive recurrent. Explain why the function 
V(x) = 11 + x2 does not satisfy the Foster-Lyapunov stability criteria for positive recurrence, for 
any choice of the constant b and the finite set C. 


2.5. Allocation of service 
Prove the claim in Example 4a about the largest value of e. 


2.6. Opportunistic scheduling (Tassiulas and Ephremides [40]) 

Suppose N queues are in parallel, and suppose the arrivals to a queue 7 form an independent, 
identically distributed sequence, with the number of arrivals in a given slot having mean a; > 0 and 
finite second moment K;. Let S(t) for each t be a subset of E = {1,..., N} and t > 0. The random 
sets S(t) : t > 0 are assumed to be independent with common distribution w. The interpretation 
is that there is a single server, and in slot 2, it can serve one packet from one of the queues in S(t). 
For example, the queues might be in the base station of a wireless network with packets queued for 
N mobile users, and S(t) denotes the set of mobile users that have working channels for time slot 
[t,t +1). See the illustration: 
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(a) Explain why the following condition is necessary for stability: For all s C E with s 4 0, 
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(b) Consider u of the form u = (u(i,s): i € E,s C E), with u(i,s) > 0, u(i,s) = 0 if i ¢ s, and 
Vier uli, s) = 14549}. Suppose that given S(t) = s, the queue that is given a potential service op- 
portunity has probability distribution (u(i, s) : ¿€ E). Then the probability of a potential service 
at queue 7 is given by pu (u) = >>, u(i, s)w(s) for i € E. Show that under the condition (2.27), for 
some € > 0, u can be selected to that a; +€ < mlu) for i € E. (Hint: Apply the min-cut, max-flow 
theorem, given in Chapter 6 of the notes, to an appropriate graph.) 

(c) Show that using the u found in part (b) that the process is positive recurrent. 

(d) Suggest a dynamic scheduling method which does not require knowledge of the arrival rates or 
the distribution w, which yields the same bound on the mean sum of queue lengths found in part (b). 


2.7. Routing to two queues — continuous time model 
Give a continuous time analog of Examples l.a and 1.b. In particular, suppose that the arrival 
process is Poisson with rate À and the potential departure processes are Poisson with rates y and 


p2. 


2.8. Stability of two queues with transfers 
Let (A1, A2, V, 41, 112) be a vector of strictly positve parameters, and consider a system of two service 
stations with transfers as pictured. 


A ,——=| station 1 MM 


x station2 > 


Station 7 has Possion arrivals at rate A; and an exponential type server, with rate u;i. In addi- 
tion, customers are transferred from station 1 to station 2 at rate uy, where u is a constant with 
u € U = [0,1]. (Rather than applying dynamic programming here, we will apply the method of 
Foster-Lyapunov stability theory in continuous time.) The system is described by a continuous- 
time Markov process on Z? with some transition rate matrix Q. (You don’t need to write out Q.) 
(a) Under what condition on (Aq, À2, V, H1, 42) is there a choice of the constant u such that the 
Markov process describing the system is positive recurrent? 

(b) Let V be the quadratic Lyapunov function, V(21, £2) = a + z, Compute the drift function 
QV. 
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(c) Under the condition of part (a), and using the moment bound associated with the Foster- 
Lyapunov criteria, find an upper bound on the mean number in the system in equilibrium, X1 + X2. 
(The smaller the bound the better.) 


2.9. Stability of a system with two queues and modulated server 

Consider two queues, queue 1 and queue 2, such that in each time slot, queue 7 receives a new 
packet with probability a;, where 0 < a; < 1 and 0 < ag < 1. Suppose the server is described by a 
three state Markov process, as shown. 


cares 
queuel 7 y; y K 


r 1 
: | 0- server longer] f 
L 


y fi i 
LTA 
. @) 


queue 2 


If the server process is in state i for i € {1,2} at the beginning of a slot, then a potential service 
is given to station i. If the server process is in state O at the beginning of a slot, then a potential 
service is given to the longer queue (with ties broken in favor of queue 1). Then during the slot, 
the server state jumps with the probabilities indicated. (Note that a packet can arrive and depart 
in one time slot.) For what values of az and az is the process stable? Briefly explain your answer 
(but rigorous proof is not required). 
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Chapter 3 


Queues with General Interarrival 
Time and/or Service Time 
Distributions 


3.1 The M/GI/1 queue 


An M/GI/1 queue is one in which the arrival process is memoryless (in fact, Poisson, with some 
parameter A > 0), the service times X1, X2,... of successive customers are independent with a 
general probability distribution function B, and there is a single server. The number of customers 
in the system at time t is denoted N(t). A sample path of N is illustrated in Figure 3.1. For ease of 
notation, it is assumed that customers are served in the order in which they arrive. This is known 
as the First-Come, First-Served (FCFS) service discipline. As indicated in the figure, we define the 
following auxiliary variables: 


Cn denotes the nth customer 

dn is the number in the system just after C,, departs, equilibrium distribution (dz : k > 0) 
q”, is the number in the system just before C,, arrives, equilibrium distribution (rg : k > 0) 
Un is the number of customers which arrive while C,, is being served 


Our goal will be to compute the distribution of N(t) in equilibrium. 

Note that the random variable N(t) for a fixed time t does not determine how long the customer 
in the server (if any) at time t has been in the server, whereas the past history (N(s) : s < t) does 
determine this time-in-service. The time-in-service of the customer in the server is relevant to 
the distribution of the future of the process N, unless the service time distribution is exponential. 
Thus, unless the service time distribution is exponential, (N(t) : t > 0) is not Markov. 

Suppose for now that equilibrium distributions exist. Since the arrivals form a Poisson sequence, 
the equilibrium distribution of N is the same as the distribution r = (rz) seen by arrivals. Since as 
most one arrival and one departure can occur at a time, r = d (see exercise). Thus, rk = pr = dk 
for all k. 

Consider the evolution of (qn : n > 0). Just after C,41 begins service, there are qn + I¢q,=0} 
customers in the system. The term / {qn=0} accounts for the arrival of Cn+1 in case the system is 
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Figure 3.1: A typical sample path for an M/GI/1 queueing system 


empty upon the departure of Cn. During the period C,,+1 is served, there are v,+1 arrivals. This 
leads to the recursion 
qn+1 = qn — Lig, Z0} + Un+1 (3.1) 


The variables v, are independent and identically distributed with common z-transform 


AT 
VEJS ER”] = . AT dB(x) (3.2) 
5 I eM 4 B (sn) (3.3) 
0 
2 BINS), (3.4) 


and mean V’(1) = —\B*’(0) = AX = p. Thus, the sequence (qn : n > 0) is Markov with the one 
step transition probability matrix (letting (az : k > 0) denote the distribution of the vn): 


ago QI ag... 
ago QI ag... 


P= ao % 0 al 


(3.5) 


Due to the nearly upper triangular structure of the matrix P it is possible to recursively find 
the elements of p as a multiple of py, and then find py at the end by normalizing the sum to 
one. Another approach is to work with z-transforms, writing the equation p = pP as zP(z) = 
[P(z) + po(z — 1)]V (2). Solve this to obtain 


(3.6) 


To determine the constant po use the fact that lim,—1 P(z) = 1 (use l'Hospital's rule and the fact 
V'(1) = AX = p) to obtain pp = 1 — p. Therefore, 


pto) = AE WHE, cn 
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which is the celebrated formula of Polleczek and Khinchine for the M/GI/1 system. Computation 
of the derivatives at one of P(z), again using l’Hospital’s rule, yields the moments N* of the 
distribution of the number in system. In particular, 


A? 
2(1 — p) 
To obtain the mean number in the queue, simply subtract the mean number in the server, p. To 


obtain the mean waiting time (in queue or system) apply Little’s law to the corresponding mean 
number in system. For example, the mean waiting time in the queue is 


p+ (3.8) 


AX? 
We (3.9) 


It is remarkable the W depends only on A, X and X2 = (X)?+Var(X). Note that W is increasing 
in the variance of X, for fixed mean X and arrival rate A. 

A simple alternative derivation of (3.9) is based on renewal theory. Suppose the order of service 
is first-come, first-served. The mean waiting time W of a typical arrival is the same as the mean 
amount of work in the system at the time of arrival, which by the PASTA property is the same 
as the mean amount of work in the system in equilibrium. The mean work in the system can be 
decomposed to the mean work in the server plus the mean work in the queue. Thus, 


W= R4+N°X, (3.10) 


R=p €3 (3.11) 


which is the product of the probability the server is busy (p) times the mean residual service time 
of a customer in service, which is determined by renewal theory. The other term on the right of 
(3.10) accounts for the portion of delay caused by the customers that are in the queue at the time 
of an arrival. Applying Little’s law, NQX = \WX = pW, substituting into (3.10) and solving 
yields W = R/(1 — p), which is equivalent to the desired formula (3.9). 


where 


3.1.1 Busy Period Distribution 


Call the customers that arrive during the service interval of a given customer an offspring of that 
customer. The offspring of a customer, together with the offspring of the offspring, and so forth, 
is the set of descendants of the customer. The length of the busy period initiated by the first 
customer is equal to the sum of the service times of the customer itself and of all its descendants. 
The number of offspring of the first customer is vı, which, given that Xı = x, has the Poisson 
distribution of mean Az. 

The length of the busy period L can be expressed as the sum 


L = X; + Li + La +... + Lu (3.12) 


where Xj, is the service time of the first customer in the busy period, vı is the number of offspring 
of that customer, and L; is the sum of the service time of the ith offspring of C1 and the service 
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time of that offspring’s descendants. Notice that the joint distribution of Lj, Lo,..., Ly, is as if 
independent samples with the same distribution as L are drawn. 
Condition on Xy to obtain: 


G*(s) = Ele] = f Sos (È neat") dB(x) (3.13) 
n=0 : 


= / e7 ST ¿ZALRAG* (92 B(dx) (3.14) 
0 
= B*(s+—AG*(s)) (3.15) 


Obtain gı = X/(1—p). Can also get this by a renewal argument. Obtain also go = X?/(1— p)?. 


3.1.2 Priority M/GI/1 systems 


Suppose there are n independent arrival streams, each modeled as a Poisson process, with the ¿th 
having arrival rate A;. Suppose the streams are merged into a single-server queueing system, and 
that the service times of customers from the ith stream are identically distributed with the same 
distribution as a generic random variable X;. 

An issue for the system may be the service order. For example, for a pure priority service 
order, the server attempts to serve the available customer with the highest priority. Some finer 
distinctions are possible. If, for example, service of a lower priority customer is interrupted by 
the arrival of a higher priority customer, the service discipline is said to be preemptive priority. 
Otherwise it is nonpreemptive priority. 

Does the mean waiting time of a customer (averaged over all the streams according to arrival 
rates) depend on the service discipline? The answer is yes, which is why mass service facilities (such 
as supermarket checkout lines) often give priority to customers with short service requirements. 

The stream obtained by merging the arrival streams is again M/GI/1 type. The arrival rate 
of the combined stream is A = A, + ... + àn, and the service time of a customer in the combined 
stream is distributed as the random variable X = Xy. Here the index J is chosen at random, with 
P[J = j| =Aj/A, and J, X1,...,Xn are mutually independent. Note that 


Nx? 
A 


i 


+ AX ao 
X= —— and X? = 3.16 
As long as customers are served in FCFS order, (3.9) is valid. The PASTA property and the FCFS 
service order implies that W is equal to the mean work in the system, where work is measured in 
units of time. Therefore, 
AX? R 
Mean work in system = = a (3.17) 
21=p) 1-p 


where R is given by (3.11), which here can be written as 


AX? Se 
2 2 E 


R= 


Although (3.17) was just derived under the assumption that the service order is FCFS, the terms 
involved do not depend on the order of service, as long as the server remains busy whenever there 
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is work to be done (in which case the system is said to be “work conserving”). Thus, (3.17) is valid 
for any work-conserving service discipline. 

Assuming the required equilibria exist, the mean work in the queue composed of type i customers 
is N;X; = A¡W,¡X; = p;W; for each i, and the mean amount of work in the server is R. The mean 
work in the system is thus ()>;_, p¡W;) + R. Equating this with the last expression in (3.17) yields 
that 


On pWi)+R= 1-2 (3.18) 


which can be simplified to yield the “conservation of work” equation: 


Y piWi = a (3.19) 
=p 

This equation is true for any work-conserving service order for which the required equilibrium 

means exist. 

In order to derive the mean waiting time of customers of stream ¿ a particular service order 
must be specified. As mentioned above, if the order is FCFS, then the mean waiting time is the 
same for all streams, and is given by (3.9). Suppose now that the nonpreemptive priority discipline 
is used, with stream 7 having priority over stream j if i < j. Let W; continue to denote the mean 
waiting time in queue for customers of stream 7. Upon arrival to the queue, a customer from stream 
1 need wait only for the completion of the customer in service (if any) and for the service of other 
stream 1 customers, if any, already in the queue. Thus, Wy = R+(A1W1)/p1, where by Little’s law 
we write AW, for the mean number of stream one customers in the queue, and R is the expected 
residual service time of the customer in service, if any: 


AX? n AX? 


Solving for W4 yields 
R 


Wı = ; 
i l- 


(3.21) 


Next, argue that Wa is equal to R + (AM1 W1)/u1 + (A2W3)/pa + A X1 W2. The first three terms 
in this sum account for the residual service time and the service times of stream 1 and stream 2 
customers already in the queue at the time of arrival of a stream 2 packet. The last term accounts 
for the expected wait due to type 1 customers that arrive while the typical arrival from stream 2 
is in the queue. Solve for Wa to find 


R W: R 
me = 2 ERRE e (3.22) 
Le pies. pi p1- p2) 
Finally, continuing by induction on +, obtain 
R 
W; = 3.23 
e a) ERA aes) 


for nonpreemptive priority order of service. 
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Figure 3.2: A GI/M/1 queue sample path 


3.2 The GI/M/1 queue 


A GI/M/1 queue is one in which the arrival process is a renewal process, service times are inde- 
pendent and exponentially distributed with a common mean, and there is a single server. Let A 
denote the cumulative distribution function and 1/A the mean of the interarrival times, and let u 
denote the parameter of the service times. The number of customers in the system at time t is 
denoted N(t). A sample path of N is illustrated in Figure 3.2. It is assumed that customers are 
served in FCFS order. 

It is useful to think of the service process as being governed by a Poisson process of rate 
u. Jumps in the process denote potential service completions. A potential service completion 
becomes an actual service completion if there is at least one customer in the system at the time 
of the potential service completion. By the properties of the Poisson process and the exponential 
distribution, this view of the system is consistent with its distribution. Define the following auxiliary 
variables, which are indicated in Figure 3.2: 


Cn denotes the nth customer 
q, is the number in the system just before Cn arrives, equilibrium distribution (rẹ : k > 0) 


Ôn is the number of potential service completions which occur during the interarrival period just 
before the arrival of Ch. 


Note that the random variable N(t) for a fixed time t does not determine how long before time 
t the last customer arrival occurred, whereas the past history (N(s) : s < t) does determine the 
time-since-most-recent-arrival. The time-since-most-recent-arrival is relevant to the distribution of 
the future of the process N, unless the interarrival distribution is exponential. Thus, unless the 
arrival process is Poisson, (N(t) : t > 0) is not Markov. Nevertheless, the mean waiting time, and 
in fact the waiting time distribution, can be determined by focusing on the process (qj, : n > 1). 

Consider the evolution of (q/, : n > 0). Just after C, arrives, there are q/, + 1 customers in the 
system. There are no more arrivals and potentially as many as q/, departures in the period strictly 
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between arrival times of C,, and C,41. Therefore, 


dn41 = (dh +1 Ón+1)+ (3.24) 

The variables ô» are independent and identically distributed with common z-transform 4*(u— 

uz). and mean —uA*(0) = u/A = 1/p. Thus, the sequence (qj, : n > 0) is Markov with the one 
step transition probability matrix (letting (8p : k > 0) denote the distribution of the ôn): 


| 1— 6o Bo 0 0 
1— bo — fi Bi o 0 


P=|1-fB0-Bi-B B B Bo 0 al (3.25) 
The equilibrium equation r = rP can be written 
Tk = bork-1 + Éire +.. k21 (3.26) 
Seek a solution of the form rg = (1 — ojo" for some ø > 0. This yields the equation for ø: 
o = A*(u — uo) (3.27) 


The function A*(u— uo) is strictly convex, value one and derivative 1/p at o = 1. Thus, a solution 
o with 0 < o < 1 exists if and only if p < 1. 

If an equilibrium probability distribution r = (rẹ : k > 0) of (q) exists, then the process is 
positive recurrent. Recall from the proof of Proposition 1.7.2, that for a given state, say k, the 
mean number of visits to state k + 1 per excursion from state k is rk+1/rpķ. Since the process can 
increase by only one at a time, in order that the process reach k + 1 before k given that it starts in 
k, it must take its first jump towards state k + 1. As soon as the process crosses below state k, it 
has no more opportunities to hit k + 1 before reaching k. By the shift invariance of the transition 
probabilities, it follows that the ratio rg+1/rķ is the same for all k. 

== = E[number of visits to k + 1 before return to k|q) = k] (3.28) 
k 
This ratio does not depend on k. Thus, if any equilibrium distribution exists, it must be a geometric 
distribution. 

Given qj, = k, the waiting time in queue of C,, is the sum of k independently distributed Poisson 
random variables with parameter u, which has Laplace transform /(u + s). Thus, the Laplace 
transform of the waiting time in equilibrium is 


oo k 
Wi _ k H 
Ble" | a 0259 (: a -) (3.29) 
a 

s+ p(l—o) 
Thus, the equilibrium waiting distribution is a mixture of the random variable that is identically 
zero, and an exponentially distributed random variable with parameter (1 — o). Thus, P[W > 
x] = oexp(—ap(1—a)). and W has mean W = 0/((1 — 0)u). By Little’s law, the mean number 
in the system is given by N = \(W + 1/4) = p/(1 — c). The total time in the system, T, can 
be represented as T = W + X, where W is the waiting time, X is the service time, and W and 
X are mutually independent. The Laplace transform of the system time is given by Ele] = 
Ele-*”)] E[e-**] = oe. That is, the total time in the system T is exponentially distributed 
with parameter u(1 — 0). 


loto (3.30) 
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3.3 The GI/GI/1 System 


A GI/GI/1 queue is one in which the arrival process is renewal type, the service times of successive 
customers are independent and identically distributed, and there is a single server. It is assumed 
that customers are served in FCFS order. 


Cn denotes the nth customer 

£n is the service time required for the nth customer 

tn is the time between the arrival of C,_1 and Cn 

wn is the waiting time in queue of the nth customer 

U(t) is the work in system, also known as the virtual waiting time, at time t 


Take as the intitial condition for the system that it is empty at time 0 and C4 arrives at time tı. 
With these definitions, notice that wn = U (ti +...+tn—). 

A useful stochastic equation is derived next. Just after the arrival of customer Cn, the amount 
of work in the system is Wn + £n, and the work begins to decrease at unit rate. Customer Cn+1 
arrives to find the system nonempty if and only if Wn + £n — tn+1 > 0, and if this condition holds 
then Wn+1 = Wn + £n — tn+1. Therefore, in general, 


Wn+1 = (Wn + Un) + (3.31) 


where Un = Tn — tn+1. The random variables of the sequence (un : n > 1) are independent and 
identically distributed, because the same is true of the two independent sequences (£n : n > 1) and 
(tn : n > 1). Then w = 0, and by repeated use of (3.31), 


Wn = Mmaxí0,w,-1 +Un-1) 
= maxf0, u, 1 + max{0, Wn—2 + Un—2}} 


= max{0, Un 1, Un—1 + Un—2 + Wn, 2} 


= max{0, Un-—1, Un—1 + Un—2, Un—1 + Un—1 + Un—2 + Wn—3} 


= max{0, Un ty Un—1 + Una tga +... u1} 


Let us next consider the question of stability. Define a random variable wy by 
Wn = max{0, u1, U1 +U49,...,41 +...Un-1), (3.32) 


which is the same as wp, but with the sequence (u1,...,Un—1) reversed in time. Since time reversal 
preserves the distribution of the u,, sequence, it is clear that for each n, w,, has the same distribution 
as Un. The sequence Ùn is nondecreasing in n, so that the limit 
Wo = lim Wy 
n—00 


= max{0, u1, u1 +u3,...) 
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exists (with values in (0, 0o]) and is equal to the maximum of the partial sums of the un sequence as 
shown. Since convergence with probability one implies convergence in distribution, it follows that 
the variables (0,,) converge in distribution to Wa. Since convergence in distribution is determined 
by marginal distributions only, it follows that 


lim-in-distn_—.ooWn = Woo (3.33) 


Suppose that the x7, and tn have finite means, so that un also has a finite mean and the strong 
law of large numbers applies: limn—oo(ui+...+Un)/n = Elu¡] with probability one. If Efun] < 0, it 
follows that limy—.o9(ui+...+Un) = —oo with probability one, which implies that Plt. < oo] = 1. 
The arrival rate is A = 1/E[ti] and the service rate is y = 1/Elx1]. Letting p = A/p as usual, 
it has just been proved that if p < 1, then the variables w» have a limiting distribution with no 
probability mass at +00. 

There are techniques for computing the distribution of waiting time. One is based on Wiener- 
Hopf spectral factorization, and another on certain identities due to F. Sipitzer. However, some 
simple bounds on the waiting time distribution, presented in the next section, are often more useful 
in applications. 


3.4 Kingman's Bounds for GI/GI/1 Queues 


The elegant bounds, due to J.F.C. Kingman, on the mean and tail probabilities for the waiting 
time in a GI/GI/1 system are given in this section. Let o? denote the variance of a random variable 
A 


Proposition 3.4.1 (Kingman’s moment bound) The mean waiting time W for a GI/GI/1 FCFS 
queue satisfies 
w< Ao? + 0?) 
2(1 — p) 


Proof. Let w denote a random variable with the distribution of wa, and let u denote a random 
variable independent of w with the same distribution as the u,'s. Taking n — oo in (3.31) yields 


(3.34) 


6 


that w £ (w+u)+, where d» denotes equality in distribution. In particular, Elw] = El(w+u)y] 
and E[w?] = E[(w + u)2]. Clearly 


w+u=(w+u)y—(w+u)-, (3.35) 


where (w + u)- = —min{0,w + u}. Taking expectations of each side of (3.35) yields Efu] = 
—El(w + u)_]. Squaring each side of (3.35), using the fact (w + u)+(w + u)- = 0 and taking 
expectations yields 


E(w") + 2E[w] E[u] + Elu”] = El(w+)J4] + El(w +1), 


or equivalently 


Efu] 


—2 E[u] 
AN 
= Ou O(w+u)- 
—2E|ul 


Noting that o? > 0 and 0? = 0? + 0? yields the desired inequality. E 
(w+u)— u x t 


Exercise: Compare Kingman’s moment bound to the exact expression for W in case the inter- 
arrival times are exponentially distributed. Solution: Then a? = > so the bound becomes 


Moz + 32) 
2(1 — p) 


whereas by the Polleczek and Khinchine mean formula for an M/GI/1 queuing system 


Elw] < 


Moz + z) 


UNE 


Next Kingman’s tail bound is given. Let wn denote the waiting time in queue of the nt” customer 
ina GI/GI/1 system. Then w, +1 has the same probability distribution as maxo<k<n U1 + ..-+Uk, 
where uz is the service time required by the kt”? customer minus the length of time between the 
kt” and k+1" arrivals. By assumption, the random variables u1, uz, ... are identically distributed. 
Suppose Elu,] < 0 and that ¢(@) = Elexp(0u,,)] < +00 for some 0 > 0. Let 0* = supí0 : p(0) < 1). 


Proposition 3.4.2 (Kingman’s tail bound) Forn > 0 and b > 0, Plwn+1 > b] < exp(—0*D). 


Proof. Choose any 9 > 0 such that p(0) < 1. Let Mo = 1 and Mp =expl0(u1 +... +ux)). Then 
the probability in question can be expressed as P[w, +1 > b] = Pl(maxo<k<n Mx) > exp(0b)]. It is 


easy to check that E[My+1|My, ..., Mo] < My, so by Lemma 3.4.3 below, 
P| max My, > exp(0b)] < E[Mo] / exp(0b) = exp(—0b). (3.36) 
SAAN 
Thus, Plw,+1 > b] < exp(—0b) for 0 < 6 < 6*, which implies the proposition. E 
Lemma 3.4.3 Let Mo, Mı, ... be nonnegative random variables such that E[Mx+11Mx, ..., Mol] 


< 
My fork > 0 (i.e. M is a nonnegative supermartingale). Then for any n > 0, P|(maxo<k<n Mp) > 
y] < E[Mo]/7- 


Proof. Let 7 = min{k > 0 : M, > y}, with the convention that T = +00 if the indicated set is 
empty. Then (maxo<k<n Mx) > y if and only if Myan > y, where a^b denotes the minimum of a and 
b. This fact and the simple “Markov inequality” implies that P[(maxo<r<n Mp) > y) = P{Mran > 
y] < E|Mran]/y. Since E[M1m] = E[Mo] for n = 0, the proof can be completed by showing that 
E| M-an] in nonincreasing in n. To that end, note that M,a(n41) Mem = (Mn41 — Mn) Lir>ny> so 


E(M, (nga) — ElMenn] = ElMar — Mn) Iron}! 
= ELE|(Mns1— MnMHir>ny| Mn, - - - , Mol] 
= E| E| (Mn+ = Mn)|Mn, e. , MolļItr>n}] 
< 0, 
where we use the fact that the event (7 > n} is determined by Mo, . . . , Mn. The lemma is proved. 
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Figure 3.3: A distribution function and its inverse 


Exercise: Compare Kingman's tail bound to the exact tail probabilities in case the service time 


distribution is exponential. Solution: In that case p(0) = ¿A (0) so the equation for 0* becomes 


A*(0) = 1 -— e With the change of variable 6 = (1 — 0) this equation becomes the same as 


the equation for o in the analysis of a GI/M/1 queueing system. Thus, 0* in Kingman's bound 
is equal to u(1 — o) where ø is the value determined in the analysis of a GI/M/1 system. Thus, 
Plwo > b] = o exp(-0*b), showing that Kingman’s bound gives the same exponential behavior as 
the actual tail probabilities. 


3.5 Stochastic Comparison with Application to GI/GI/1 Queues 


Before describing the most basic part of the theory of stochastic comparison theory, we shall review 
the problem of generating a random variable with a given cumulative distribution function. Suppose 
F is a probability distribution function and that U is a random variable uniformly distributed on 
the interval [0, 1]. The inverse function of F is defined by 


FA (p) = sup{ax : F(x) < p} (3.37) 


and an example is pictured in Figure 3.5. The graph of F~! is the graph of F reflected about 
the line of slope one through the origin. Let X = F~'(U). Then for any constant c, P[X < c] = 
P[F-\(U) < e] = P[U < F(c)] = F(c), so that X has distribution Fy. 

Exercise 1. a) Find and sketch F~! assuming that F is the distribution function of an exponential 
random variable with parameter a. b) Repeat part a) in case F corresponds to a random variable 
uniformly distributed over the set {1,2,3,4,5,6}. Simultaneously sketch F and F~! as is done in 
Figure 3.5. Is F7! right-continuous? c) Show that if F(c) < G(c) for all c, then F7*(p) > G7*(p) 
for all p. 


Definition 3.5.1 Let F and G be two distribution functions. We say F stochastically dominates 
G, and write F > G, if F(c) < G(c) for all constants c. If X and Y are two random variables, 
we say that X stochastically dominates Y, and write X > Y, if Fx stochastically dominates Fy, 
where Fx is the distribution function of X and Fy is the distribution function of Y . 
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Note that whether X > Y, depends only on the individual distributions of X and Y, and 
otherwise does not depend on the joint distribution of X and Y. In fact, X and Y don't even need 
to be random variables on the same probability space. 

The fundamental proposition of stochastic comparison theory is the following: 


Proposition 3.5.2 X > Y if and only if there is a pair of random variables (XY) defined on 
some probability space such that the following three conditions hold: 


x EX (the notation «£ means the variables have the sam probability distribution.) 
yy 
P[X > Y] =1. 


Proof. If X > Y, let U be a uniform random variable and let X = Fxy'(U) and Y = Fy'(U). 
Then (X,Y) satisfies the required three conditions. 

Conversely, if there exists (X, Y) satisfying the three conditions then for any constant c, the 
event {X < c} is a subset of the event (Y < c} (possibly ignoring an event of probability zero). 
Therefore, Fx(c) = Fz(c) < Fy(c) = Fy (c), so that indeed X > Y. a 


A useful method for deriving bounds on probability distributions, called the stochastic compar- 
ison method, or coupling method, consists of applying Proposition 3.5.2. Note that the random 
variables X and Y may have little to do with each other. They may be independent, or even defined 


on different probability spaces. Variables X and Y defined on a common statespace, with a 


and Y £ Y are called coupled versions of X and Y. Thus, to establish that X > Y, it suffices to 
show the existence of coupled versions X and Y such that P[X > Y] =1. 


Example Let w,,, for n > 1, denote the sequence of waiting times for the customers in a 
GI/GI/1 queue, which is empty at the time of arrival of customer one. Then the random variables 
101,2, ... defined by (3.32) are coupled versions of w1, we,.... Since the random variables Ùn are 
nondecreasing in n with probability one, conclude that w¡ < wa <---. That is, the waiting time 
of customer Cn stochastically increases with n. 


Example As a related example, let us demonstrate that the waiting time in a GI/GI/1 queue 
is an increasing function of the service time distribution in the sense of stochastic comparison. 
That is, suppose there are two queueing systems, called system 1 and system 2. Suppose that both 
systems are initially empty and that the sequence of arrival times of the two systems have identical 
distributions. Suppose that both systems use the FIFO service order and that the service times of 
the customers in system 7, (i = 1 or i = 2) are independent and identically distributed with CDF 
B®. Let mo denote the waiting time of the nt” customer in system 1. 

Suppose that B®) > BG), We will establish that for each n, wi > we. Consider a probabil- 
ity space on which there is defined a sequence of variables (T1, T3,...) which has the same distribu- 
tion as the sequence of interarrival times for either of the two queuing systems being compared. On 
the same probability space, for each n, let (x x?) be a pair of random variables such that x 
has CDF B®, x?) has CDF B®), and px > x?) = 1. Such a pair can be constructed accord- 


ing to Proposition 3.5.2. Moreover, the pairs in the sequence (x, ON Ve can be constructed 
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to be mutually independent, and also to be independent of (T1, T2,...). For i = 1 or i = 2, define 
we to be the waiting time of the nt” customer in the queueing system in which (T1, T>,...) is the 
interarrival time sequence and (B®, B®, ...) is the sequence of customer service times. Clearly 
WO and wo have the same distribution, so it suffices to prove that PIO > We) = 1. How- 
ever, the fact we = we = 0, the update equations wa, =a j AS —=Tn+1 + KO Vie and the fact 
px > x?) = 1 imply that PIO > Wo = 1 for all n > 1 by induction on n. Applying 


Proposition 3.5.2 again, we conclude that wo > we for all n, as advertised. 


Execise 2a) Show that X > Y if and only if for every nondecreasing function ¢, E[¢(X)| > ElA(Y)]. 
(Another type of stochastic ordering requires the inequality to hold only for convex, nondecreasing 
functions ¢.) b) Let X be uniformly distributed on [0,1], let Y be exponentially distributed with 
E[Y] = 1, and let Z have the Poisson distribution with E[Z] = 1. Identify any stochastic orderings 
that appear between pairs of these random variables (e.g., is X > Y?). c) Suppose X > Y and 
E|X] = E[Y]. Prove that X and Y have the same distribution. 


3.6 GI/GI/1 Systems with Server Vacations, and Application to 
TDM and FDM 


In a GI/GI/1 system with server vacations, whenever the server finishes serving a customer or 
returns from a vacation, and finds the system empty, the server goes on another vacation. A 
vacation is a period of time in which the server does not serve customers, even if one or more 
customers arrive during the period. An effective method for accounting for the effect of vacations 
is to compare the sample paths of a system operating without vacations to one with vacations. The 
following variables will be used. 


e w? is the waiting time in queue of the nt? customer in the system without vacations. 
e wn is the waiting time in queue of the nt” customer in the system with vacations. 


e t; is the time of the first arrival and, for n > 1, tn41 is the time between the nt? and n+ 1” 
arrivals. These variables are used in both systems. The variables t1,t2, .. . are assumed to 
be independent and identically distributed. 


e xn is the service time required by the nt” customer. These variables are used in both systems. 
The variables 11,12, . . . are assumed to be independent and identically distributed. 


e V, is the length of the nt”? vacation period for the system with vacations. The variables V, 
for n > 1 are assumed to be independent, and identically distributed. 


The two systems both start empty, and in the system with vacations, the server initiates a 
vacation at time zero with duration Vo. Roughly speaking, the main difference between the two 
systems is that when the nt? customer begins service in the system with vacations, the total idle 
time of the server is of the form Vo + ... + Vk for some k. In fact, k is the minimum integer, 
subject to the constraint that the idle time in the system with vacations is greater than or equal to 
the idle time in the system without vacations. This idea is expressed concisely in the lemma below. 

Let y(u) denote the residual lifetime process for the renewal process determined by Vo, Vi, ... . 
That is, y(u) = Vo + ... + Viu) — u, where l(u) = min{k : Vo+ ... +V_ > u}. Also, let I? 
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denote the total idle time, in the system without vacations, up to the time the nt? customer begins 
(or ends) service. 


Lemma 3.6.1 Forn > 1,0, = we + y(IR). 


Proof. Argue by induction on n. Since w; = 7(J?) and w? = 0, the lemma is true for n = 1. 
Suppose the lemma is true for n, and argue that it is true for n + 1 as follows. Let x denote the 
amount of time (possibly zero) the server in the system without vacations is idle between services 
of the nt” and n + 1 customers. Therefore, [?,, = I} + x, and two cases are considered: (1) If 
x < Wn — wy, then then the server in the system with vacations will not take a vacation between 
service of the nt” and n +1" customers. Therefore wn41 — W241 = Wn — W8 — x = y(1p)- x = 
vr +x) = yUR41), as was to be proved. (2) If x > wn — wp then at the time of arrival of the 
n + 1% customer, both systems are empty and both have been idle for 12 +1 time units. Therefore, 
Wat = Y(Li+1) and 17,1 = 0, so again wn+1 — W41 = yU7,,). The induction step is complete, 
and the lemma is proved. E 


By renewal theory, if the distribution of V; is nonlattice type, the random variable y(u) converges 
in distribution as u — oo, where the limit has probability density function (p.d.f.) fy given by 


P[V > d 
f(c) = TA (3.38) 
It is useful to skip the question of convergence to equilibrium by choosing the distribution of Vo 
to make (y(u) : u > 0) a stationary random process (rather than only asymptotically stationary). 
Thus, assume that Vo has p.d.f. f,. Consequently, y(u) has distribution function f, for all u > 0. 
Since (y(u) : u > 0) is independent of (w2, 12), the conditional density of 7(I°) given (w2, 19) is 
again fy. Thus (wy, Tp) and y(£;) are mutually independent. The following proposition is proved. 


Proposition 3.6.2 Suppose Vo has p.d.f. fy. Then ws, and y(I?) are independent, y(I}) has p.d.f. 
fy, and wn = w? + (12). 


Since the kt” moment of a random variable with p.d.f. fy is mp4i/((k + 1)m1 where mj = E[Vf, 
the following corollary is immediate from the Proposition. 


Corollary 3.6.3 Suppose Vo has p.d.f. fy. Then Elw,] = Elw;] + m2/(2m1) and Var(w,) = 
Var(w2) + m3/(3m1) — (ma2/(2m1))?. 


If Vo has some arbitrary distribution, the Proposition still holds in a limiting sense, under an 
additional assumption to prevent periodic behavior. For example, if E[a1] < E[ti] < oo and if Vj 
or xı — tı is nonlattice type, then the pair (y(12), w2) converges in distribution as n > oo, and the 
limit (7, w) has independent coordinates, with the first having p.d.f. fy. 

As an example, one can model a time-slotted system in which the time axis is divided into 
frames of length M, for some constant M, and service can only begin at the beginning of a frame. 
To that end, choose the service time random variable x; to be a possibly-random integer multiple 
of M, and suppose that the vacations are all M units in duration. Then fy is the uniform density 
on the interval [0, M]. By the corollary, the mean waiting time in the system is M/2 time units 
larger than the mean waiting time in the system with the same service time distribution, without 
the constraint that service begin at the beginning of frames. 
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Continuing this example further, if M is an integer, we can think of a frame as being composed 
of M unit length slots. The waiting time in the slotted system is the same as that for a stream 
traversing a Time Division Multiplexed (TDM) communication link, in which the stream is served 
during one slot per frame. In the TDM system there are M arrival streams, all multiplexed onto a 
single line, and the queuing processes for the M streams are not related. When the vacations are 
dropped, the model represents a Frequency Division Multiplexed (FDM) communication link of the 
same total capacity. In an FDM multiplexed link, the M streams again do not interact. The mean 
waiting times are thus related by Wrpm = Wrpm + M/2. However, for the TDM system, the 
actual service time of a packet is M — 1 time units less than the service time in an FDM system, so 
the total mean system time for packets in the two systems are related by Trpm = Trpm —M/2+1. 
Note that there is not much difference between the delay in TDM and FDM systems, especially for 
heavily loaded systems. 

Finally, to specialize the example further, if the traffic arrival process is Poisson with rate A/M 
packets/slot and if each packet is served in M slots in the FDM system and in one slot in the TDM 
system (with only one out of every M slots being available), then the FDM system is an M/GI/1 
system with constant service times (i.e. an M/D/1 system). The P-K mean formula thus yields 
Wrpm = AM/(Q(1— A)), and so Wrpm = Wrpm + M/2 = M/(2(1 — A)). 


3.7 Effective Bandwidth of a Data Stream 


There has been an extensive effort since the inception of packet-switched communication networks 
to characterize the traffic carried by networks. The work aims to account for the bursty nature of 
many data sources. This section describes one approach for defining the effective bandwidth of a 
data stream. The approach is based on the theory of large deviations. 

Recently, there has been a keen interest in accounting for the observations of many studies of 
traffic in real networks, which indicate that traffic streams exhibit self-similar behavior. That is, 
the random fluctuations in the arrival rate of packets appears to be nearly statistically the same 
on different time scales, ranging over several orders of magnitude. We touch on this development 
briefly in the context of the effective bandwidth of a self-similar Gaussian source. An extensive 
annotated bibliography on the subject is given in [43]. 

One of the primary goals of information theory is to identify the “effective information rate” 
of a data source. The entropy or the rate-distortion function of a data source may be thought of 
as such. The theory of effective bandwidth, described in this section, has a similar goal. Another 
connection between the theory of effective bandwidth of data streams and information theory is 
that much of the theory of effective bandwidth is based on large deviations theory, which is a close 
cousin to Shannon's theory of information. Moreover, there is possibly some yet to be discovered 
direct connection between the theory of effective bandwidth and Shannon's theory of information. 
For example, perhaps an “effective-bandwidth vs. distortion” function can be computed for some 
nontrivial sources. 

A major way that the theory of effective bandwidth differs from the Shannon theory is that it 
treats the flow of data bits as it would the flow of anything else, such as a fluid. The values of the 
bits are not especially relevant. The idea is that individual connections or data streams carried by a 
network may be variable in nature. The data rate may be variable in a statistical sense, in that the 
rate produced by the stream or connection is not a priori known and can possibly be considered to 
be random. It may also be varying in time: such is the case for data streams produced by variable 
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rate source coders. Suppose many variable data streams are multiplexed together onto a line with 
a fixed capacity, where capacity is measured in bits per second. Because of statistical multiplexing, 
the multiplexer has less work to do than if all the data streams were sending data at the peak rate all 
the time. Therefore a given data stream has an effective bandwidth (which depends on the context) 
somewhere between the mean and peak rate of the stream. The use of the word “bandwidth” in the 
phrase “effective bandwidth” has a different meaning than that commonly used by radio engineers. 
A better name might be “effective data rate.” However, the name “effective bandwidth” is already 
deeply entrenched in the networking literature, so is used here. 

To illustrate the ideas in the simplest setting first, we begin by considering a bufferless com- 
munication link, following Hui [20, 21]. The total offered load (measured in bits per second, for 
example) is given by 


J Nj 
XSS Y Xy (3.39) 
j=1 i=1 
where J is the number of connection types, n; is the number of connections of type j, and Xj; is the 
data rate required by the ¿th connection of type j. Assume that the variables Xj; are independent, 
with the distribution of each depending only on the index j. If the link capacity is C then the 
probability of overload, P[X > C], can be bounded by Chernoff’s inequality: 


J 
log P[X > C] < log Ee**—] = s(Y nja;(s) — C) (3.40) 
j=1 
where a;(s) is given by 
1 
aj(s) = E log Ele*%;:]. (3.41) 
Thus, for a given value of y, the quality of service constraint log P[X > C] < —7 is satisfied if the 
vector n = (m1,...,7 7) lies in the region 
J 
A= {ne RẸ minls(nay(s) = C)] < 1) = UAC) (3.42) 
TS 
where 
J 
A(s) = {n€ R]: S njaj(s) <C- Lyi (3.43) 


j=l : 
The complement of A relative to RŽ is convex. Let n* be on the boundary of A (think of n* as 
a “nominal” value of the vector n). A polyhedral subset of A, delineated by a hyperplane tangent 
to the boundary of A at n*, is given by A(s*), where s* achieves the minimum in (3.42). Thus, any 


vector n € Z 3 satisfying 
J 


S njaj(*) <C- = (3.44) 
j=l z 
satisfies the quality of service constraint. Once C, y, and s* are fixed, the sufficient condition (3.44) 
is rather simple. The number a;(s*) is the effective bandwidth of a type j connection, and C—y/s* is 


the effective capacity. Condition (3.44) is analogous to the condition in classical information theory 
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which insures that a particular channel is capable of conveying several independent data sources 
within specified average distortions, namely that the sum of the rate distortion functions evaluated 
at the targeted distortions should be less than or equal to the channel capacity. 

As long as the random variables Xj; are not constant, the function a; is strictly increasing, 
and ranging from the mean, E[X;;¡] as s — 0, to the peak (actually the essential supremum, 
sup{c : P[X;; > c] > 0}), of X;;. Note that the effective bandwidth used depends on the variable 
s*. Such dependence is natural, for there is a tradeoff between the degree of statistical multiplexing 
utilized and the probability of overload, and the choice of the parameter s* corresponds to selecting 
a point along that tradeoff curve. Roughly speaking, as the constraint on the overflow probability 
becomes more severe, a larger value of s* is appropriate. For example, if y is very large, then 
the sets A, are nonempty only for large s, so that the choice of s* is also large, meaning that the 
effective bandwidths will be near the peak values. 

The set A defined in (3.42) is only an inner bound to the true acceptance region for the constraint 
log P[X > C] < —y. However, if C and y both tend to infinity with their ratio fixed, then the 
true acceptance region, when scaled by dividing by C, converges to A scaled by dividing by C [23]. 
(The scaled version of A depends only on the ratio of C to y.) This follows from Cramér’s theorem 
(see [36]), to the effect that Chernoff’s bound gives the correct exponent. 

So far, only a bufferless link confronted with demand that is constant over all time has been 
considered. The notion of effective bandwidth can be extended to cover sources of data which vary 
in time, but which are statistically stationary and mutually independent [3, 37, 7]. Let X,;[a, b] 
denote the amount of traffic generated by the ith connection of type j during an interval [a, b]. We 
assume that the process X is stationary in time. Set 


1 
aj(s,t) = = 08 E[e5%+ 0:4], (3.45) 
s 
For t fixed, the function a; is the same as the one-parameter version of a; considered above, 


applied to the amount of work generated in an interval of length t. Beginning with the well-known 
representation of Loynes for the stationary queue length, Q(0) = sup;s9 X [—t, 0] — tC, we write 


log P[Q(0) > By 


= log ABP) —tC} > B] (3.46) 

~ aop log P|X|—t,0] — tC > B] (3.47) 
i J 

K ap min Z njaj(s,t) — s(B + tC) (3.48) 


The symbol “~” used in (3.47) and (3.48) denotes asymptotic equivalence of the logarithms of 
the quantities on either side of the symbol. The use of this symbol is justified by limit theorems 
in at least two distinct regimes: (1) the buffer size B tends to infinity with n and C fixed, and 
(2) the elements of the vector n, the capacity C, and the buffer space B all tend to infinity with 
the ratios among them fixed. Under either limiting regime, the line (3.47) is justified by the fact 
that the probability of the union of many rare events (with probabilities tending to zero at various 
exponential rates) is dominated by the probability of the most probable of those events. The 
line (3.48), which represents the use of the Chernoff bound as in (3.40), relies on the asymptotic 


77 


exactness of the Chernoff bound (Cramér’s theorem or more general large deviations principles such 
as the Gártner-Ellis theorem. See [36]). 

Equations (3.46)-(3.48) suggest that the effective bandwidth to be associated with a connection 
of type j is a;(s*,t*), where t* achieves the supremum in (3.48), and s* achieves the minimum in 
(3.48) for a nominal value n* of n. The approximate condition for meeting the quality of service 
requirement log P[Q(0) > B] < —y for n near n* is then 


This region scales linearly in y if n*, B and C scale linearly in y, and asymptotically becomes 
a tight constraint as y — oo. The value t* is the amount of time that the system behaves in an 
unusual way to build up the queue length just before the queue length exceeds B. The quantity 
C+ B/t* — y/s*t* is the effective capacity of the link. Following [35], we call t* the critical time 
scale. In the first limiting regime, described above, t* tends to infinity, so the effective bandwidth 
becomes aj(oo, s*). Use of the Gártner-Ellis theorem of large deviations theory allows the limit 
theorems in the first limiting regime to be carried out for a wide class of traffic streams with 
memory. 

The above approximation simplifies considerably in the case that the traffic process is Gaussian. 
In particular, suppose also that there is only one class of customers (so we drop the index j and let 
n denote the number of connections) and that for each i the traffic X;(0, t] is Gaussian with mean 
At and variance V(t). The corresponding effective bandwidth function is a(s,t) = A+ sV(t)/2t. 
Inserting this into (3.48) and then performing the minimization over s yields that 


((e — A)t +b)? 
2V (8) 


log P[Q(0) > B] ~ ninf (3.49) 
where b is the buffer space per connection (so B = nb) and cis the capacity per connection (C = nc). 

Suppose V(t) /t? converges to a finite constant 0? as t tends to infinity, where H, known as the 
Hurst parameter, typically satisfies 1/2 < H < 1. If H = 1/2, we see the process does not exhibit 
long range dependence. In particular, if X has independent increments (therefore the increments of 
a Brownian motion with drift A and diffusion parameter 07), then V(t) = o?¢ and moreover (3.49) 
holds with exact equality. 

If H > 1/2 (but still H < 1) then the critical time scale t* is still finite. That is, even in the 
presence of long range dependence, the critical time scale is still finite in the limiting regime of C, 
B and n tending to infinity with fixed ratios among them [35]. The value of V(t) for t larger than 
t* therefore does not influence the approximation. 

See [11] and [23] for extensive surveys on effective bandwidth, and [43] for a very extensive 
bibliographic guide to self-similar traffic models and their use. The paper [4] presents significant 
bounds and analysis related to notions of equivalent bandwidth with a different terminology. Fi- 
nally, the paper [22] connects the theory of effective bandwidths to thermodynamics and statistical 
mechanics. 


3.8 Problems 


3.1. A queue with customers arriving in pairs 
Customers arrive at a single-server queue two at a time. The arrival instants for pairs of customers 
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are given by a Poisson process with rate A. The service times of customers are independent, 
exponentially distributed with parameter u. Let N(t) denote the number of customers in the 
system at time t. (a) The process (N(t) : t > 0) is a continuous time pure jump Markov process 
with statespace Z+. Sketch the transition rate diagram of N. (b) See what can be deduced about 
N by applying the Foster-Lyapunov stability criteria and related moment bounds for the Lyapunov 
function V(x) = x. (c) Repeat part (b) using V(x) = r; 

(d) Solve for the z transform of the equilibrium distribution and find the equilibrium distribution. 
Under what condition does it exist (i.e. under what condition is N positive recurrent?) 

(e) Let (rp : k > 0) and (dp : k > 0) be the equilibrium probabilities defined by 


rk =  Plthere are k in the system just before the arrival of a pair of customers] 


di =  Plthere are k in the system just after the departure of a typical customer]. 


Express (rp : k > 0) and (dz : k > 0) in terms of (pz : k > 0), A and u. In particular, does rk = pk? 
or dk = Pp? or rk = dy? 


3.2. Token bucket regulation of a Poisson stream 

A token bucket traffic regulation scheme works as follows. The packet stream to be regulated is 
modeled as a Poisson stream with arrival rate A. There is a token pool that holds up to B tokens. 
When a packet arrives, if there is a token in the token pool then the packet instantly passes through 
the regulator, and one of the tokens is removed from the token pool. If instead the token pool is 
empty, then the packet is lost. (Notice that packets are never queued.) New tokens are generated 
periodically, with one time unit between successive generation times. If a token is generated when 
the token pool is full, the token is lost. (The token pool acts like a leaky bucket”.) (a) Identify an 
embedded discrete-time, discrete-state Markov process, and describe the one-step transition prob- 
abilities of the chain. (b) Express the fraction of packets lost (long term average) in terms of A, B 
and 7, where m denotes the equilibrium probability vector for your Markov chain. (You do NOT 
need to find 7.) (c) As an approximation, suppose that the times between new token generations 
are independent, exponentially distributed with common mean 1. Find a fairly simple expression 
for the loss probability. 


3.3. Extremality of constant interarrival times for G/M/1 queues 

Consider two G/M/1 queueing systems with common service rate u. The first has a general in- 
terarrival distribution function A(t) with mean 1/A such that A/u < 1. The second has constant 
interarrival times, all equal to 1/X. (Thus the interarrival distribution function in the second sys- 
tem is Ag(t) = 0 if t < 1/A and Ag(t) = 1 otherwise, and the system is called a D/M/1 system). 
(a) Show that the Laplace transforms are ordered: A%(s) < A*(s) for all s > 0. (b) Show that 
the mean number in the system at arrival times (in equilibrium) and the mean waiting time in the 
system is smaller for the D/M/1 system. 


3.4. Propagation of perturbations 

Consider a single-server queueing system which is initially empty and for which a total of five 
customers arrive, at times 1,2,5,13, and 14, respectively. Suppose the amounts of time required to 
serve the customers are 5,2,4,2,2 time units, respectively. (a) Sketch the unfinished work in the 
system as a function of time. Indicate the departure times of the five customers, and compute the 
waiting times in queue of the five customers. (b) Repeat part (a), assuming that the service time 
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of the second customer is increased to 3 time units. (c) Repeat part (a), assuming the service time 
of the second customer is increased to 4 time units. (d) Describe briefly in words how in general 
an increase in service time of one customer effects the waiting times of other customers in the system. 


3.5. On priority M/GI/1 queues 

Consider an M/GI/1 system with preemptive resume priority service. There are assumed to be two 
priority classes, with independent arrival streams and service requirements. Class 1 has priority 
over class 2. Customers of the ith class arrive according to a Poisson process of rate A;, and the 
service times of customers of class ¿ are independent and identically distributed. Use X; to denote 
a random variable with the same distribution as the service requirements for class i, for i = 1 or 
i = 2. (a) Describe what “preemptive resume priority” means. (b) Derive the mean time in the 
system, T; and To, for class 1 and class 2, respectively. 


3.6. Optimality of the uc rule 

Consider an M/G/1 queue with customer classes 1 through K and a nonpreemptive priority ser- 
vice discipline. Let u; = 1/X; where X; denotes the mean service time for type i customers and 
let c¿, with c; > 0, denote the cost per unit time of holding a class 7 customer in the queue. For 
some permutation o = (01,01,...,0K) of (1,2,..., K), the class 0, customers are given the highest 
priority, the class g2 customers are given the next highest priority, and so on. The resulting long 
run average cost per unit time is >>, A¡c¿W;(0), where W;(c) is the mean waiting time of class i 
customers under the priority order ø. (a) Show that an ordering o minimizes the long-run aver- 
age cost per unit time over all possible orderings if and only if HoiCoi > HozCoz > ++» 2 Mow Cox: 
(Hint: If o does not satisfy the given condition, then for some i, Ho;Coi < Moj+1Coi,,- Appeal to the 
conservation of work equation to argue that by swapping c; and o;41, an ordering ô with strictly 
smaller cost than ø is obtained. You still also need to figure out how to prove both the “if” and 
“only if” portions of the statement.) (b) Write a brief intuitive explanation for the result of part (a). 


3.7. A queue with persistent customers 

Consider a queue with feedback as shown, where AD > 0 and 0 < b < 1. New arrivals occur 
according to a Poisson process of rate A. The service time of a customer for each visit to the 
queue is the constant D. Upon a service completion, the customer is returned to the queue with 
probability 1 — b. 


(a) Under what condition is the system stable? Justify your answer. 

(b) Suppose the service order is FIFO, except that a returning customer is able to bypass all other 
customers and begin a new service immediately. Denote this by PR, for priority to returning, 
service order. Express the mean total system time of a customer, from the time it arrives until 
the time it leaves the server for the last time, in terms of A,b, and D. (Hint: A geometrically 
distributed random variable with parameter p has first moment z and second moment =P.) 

(c) If instead the service within the queue were true FIFO, so that returning customers go to the 
end of the line, would the mean total time in the system be larger, equal, or smaller than for PR 
service order? Would t 
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he variance of the total time in the system be larger, equal, or smaller than for PR service order? 


3.8. A discrete time M/GI/1 queue 

Consider a single server queue in discrete time. Suppose that during each time slot, one customer 
arrives with probability p, and no customers arrive with probability 1 — p. The arrival events for 
different slots are mutually independent. The service times are independent of the arrival times 
and of each other, and the service time of any given customer is equally likely to be 1,2,3,4,5 or 6 
slots. Find W, the mean number of complete slots spent in the queue by a typical customer (don't 
include time in service). Show your reasoning. Also identify an embedded Markov chain for the 
system. 


3.9. On stochastic ordering of sampled lifetimes 
Let f and g be two different probability density functions with support Ry, and with the same 
af (x) 


finite mean, mı. Let fr and gr denote the corresponding sampled lifetime densities: fr (x) = mar 


and gL(1) = = Show that it is impossible that fg < gz. (We write fr < gz to denote that a 
random variable with pdf fz is stochastically smaller than a random variable with pdf gz. Note 
that the mean of fr does not have to equal the mean of gz.) 


3.10. Effective bandwidth, bufferless link 

Consider a bufferless link of fixed capacity C = 200 serving connections of 2 types. The data rate of 
each connection is random, but constant in time. The data rate required by a connection of type 1 
is uniformly distributed over [0,2], the data rate required by a connection of type 2 is exponentially 
distributed with mean 1, and requirements for different connections are mutually independent. (a) 
Calculate the effective bandwidth functions a1(s) and a2(s). (b) Find the largest integer n so that 
the Chernoff inequality implies that the blocking probability is less than or equal to 0.001 for a 
nominal load of n connections of each type. Show your work for full credit! (Hint: Write a short 
computer program for this problem. For a given value of n the Chernoff bound is computed by a 
minimization over s. Then n can be adjusted by binary search — increasing n if the corresponding 
overflow probability is too small, and decreasing n if the corresponding overflow probability is too 
large.) (c) Compute the effective bandwidths a, (s*), a2(s*), and the effective capacity C+ ey, 
Which of the two effective bandwidths is larger? Sketch the corresponding acceptance region A(s*). 
Here s* denotes the optimizing value of s in the Chernoff bound. (Hint: The nominal load point 
(n,n) should be contained in the region A(s*). Which of the two effective bandwidths is larger?) 


3.11. Effective bandwidth for a buffered link and long range dependent Gaussian traf- 
fic 

Consider a link with buffer size B = 100 and fixed capacity C = 200, serving connections of 2 
types. The amount of data offered by a connection of type i over an interval of length t is assumed 
to be Gaussian with mean t and variance V;(t) = 124%, where H; is the Hurst parameter. Assume 
Hı = 0.5 and Hə = 0.9. (So the class 2 connections have long-range dependence.) (a) Compute 
the largest value n so that the Chernoff approximation for the overflow probability for n connec- 
tions of each type (simultaneously) is less than or equal to exp(—y) = 0.001. (Hint: Write a short 
computer program for this problem. For a given value of C and t the approximation is computed 
by a minimization over s, which can be done analytically. The maximization over t can be done 
numerically, yielding the approximate overflow probability for the given value of C. Finally, C can 
be adjusted by binary search as in the previous problem. Also, compute the effective bandwidths 
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a1(s*,t*), ag(s*,t*), the critical time £*, and the effective capacity Ceff = C + E — siz. Sketch 
the corresponding acceptance region A(s*,t*). (Hint: The nominal load point (n,n) should be 
contained in the region A(s*,t*). Which of the two effective bandwidths is larger?) (b) Redo 
problem (a) for overflow probability exp(—y) = 0.00001. and comment on the differences between 


the answers to parts (a) and (b). 


3.12. Extremal equivalent bandwidth for fixed mean and range 

Suppose X is a random variable representing the rate of a connection sharing a bufferless link. 
Suppose the only information known about the probability distribution of X is that E[X] = 1 and 
P[0 < X < 2] = 1. For a given s > 0 fixed, what specific choice of distribution of X meeting these 


y = MEN 


constraints maximizes the equivalent bandwidth a(s), where a(s ; 
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Chapter 4 


Multiple Access 


4.1 Slotted ALOHA with Finitely Many Stations 


Suppose M stations wish to communicate to each other via a shared broadcast channel. Suppose 
that time is divided into equal length slots such that the length of each slot is the amount of time 
needed for a station to transmit one packet of information. 

Consider a particular station and a particular time slot. If the station does not have a packet 
at the beginning of the slot, then it receives a new packet in the slot with probability q,. A new 
packet (one that arrives in the slot) cannot be transmitted in the slot. If the station already has a 
packet to transmit at the beginning of the slot, then it transmits the packet with probability q, in 
the slot, and the station is not eligible to receive any new packets in the slot. The events of packet 
arrival and transmission at the different stations are mutually conditionally independent, given the 
status of the stations at the beginning of the slot. 

If no station transmits a packet in the slot, the channel is said to be “idle” during the slot, and if 
two or more stations transmit during the slot, a “collision” is said to occur. If either the slot is idle 
or a collision occurs in the slot, then no packet is successfully transmitted. If exactly one station 
transmits a packet in the slot, then that transmission is called a “success,” and the packet leaves 
the system. The station making the successful transmission is eligible to receive a new packet in 
the next time slot, while all other stations with packets carry their packets over to the next slot. 

The parameters of the model are thus M,qq and qr. Several aspects of the model involve 
choices that were made largely for simplicity and tractability. For example, we might have allowed 
a station to continue receiving packets, even after it already had one to send. However, in some 
systems the successful transmission of the first packet in a queue at a station is the most crucial, 
because that packet can reserve slots (possibly on another channel) for the packets in queue behind 
it. Also implicit in the model is that a station can determine by the beginning of the next slot 
whether its transmission is successful, so that it knows whether to transmit the same packet again, 
or to switch back to the mode of receiving a new packet. One way to modify the current model 
to account for a delay of, say, R time slots would be to time-interleave R independent versions of 
the system. On the other hand, for networks of small physical size, feedback comes so soon that 
an unsuccessful transmission can be stopped well before the whole packet is transmitted. That is 
the idea of collision detect. 

A final aspect of the model we should comment on is that new packets are treated the same 
way as packets that have already been transmitted. An alternative is to transmit a new packet 
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with probability one in the first slot after it arrives, and if such transmission is not successful then 
the packet should be retransmitted with probability q, in subsequent slots, until it is successfully 
transmitted. Thus, we have adopted a delayed first transmission (DFT) rather than immediate first 
transmission (IFT) model. The qualitative behavior and exact analysis of the two types of models 
are similar. The IFT model can lead to lower mean delays for lightly loaded systems. On the other 
hand, the DFT model is less sensitive to variations in the statistics of the arrival process. 

Quantities of interest include the mean throughput and mean delay of the system. Since the 
stations are interchangeable in the model, the process (N(k) : k > 0) is a Markov process, where 
N(k) is the number of stations with packets at the beginning of a slot k. Given N(k) = n, the 
conditional probability of a successful transmission in slot k, Ps(n, qx), is given by Ps(n, qz) = 
nqz(1 — qr)”~+, and the conditional probability that there are j arrivals is b(M — n, qa, j), where 
b() is the binomial mass function 


M-n j M-=n=3 $ 
; : 1— "I O0<7<M-n 
b(M — n, qa, $) = ( j ) at ia) tine 


0 else. 


Furthermore, whether there is a successful transmission is conditionally independent of the number 
of new arrivals, given N(k) = n. Thus, the one step transition probabilities of the process can be 
expressed as follows. 


PIN(k +1) =n+i|N(k) =n] = Ps(n, qz)b(M — n, qa, i +1) + (1 — Psín, gz))b(M — n, da, 1) 


The equilibrium probability distribution 7 of the Markov process (N (k)) can be easily computed 
numerically by matrix iteration. The mean number of packets in the system and mean throughput 
are given by N = Sa g NT and S = pie o TnPs(n, dz) = (M — N)qa respectively, and by Little’s 
law the mean delay is N/S. 

The dynamical behavior of the system can be better understood by considering the drift of the 
random processes (N(k)). The drift at instant k is defined to be E[N(k + 1) — N(k)|N(k)]. It is 
a function of N(k), and is equal to the expected change in N over slot k, given N(k). Since the 
change in N is the number of arrivals minus the number of departures, the drift can be expressed 
as d(N(k)) where d(n) = (M — n)qa — Ps(n, qz). 

It is convenient to think of the function d(n) as a one-dimensional vector field. For convenience, 
consider d(n) for real values of n between 0 and M. The zeros of d(n) are called equilibrium points, 
and an equilibrium point is called stable if d(n) makes a down-crossing at the equilibrium point. 
We expect that the sample paths of the Markov process spend much of their time near the stable 
equilibrium points of (N(k)). If q, is sufficiently small for given values of M and qa, then there is 
only a single equilibrium point, which is stable. However, for some parameter values there can be 
three equilibrium points, two of which are stable. One of the stable points occurs for a large value of 
n, corresponding to large delay and small throughput, whereas the other equilibrium point occurs 
for a much smaller value of n. The existence of stable equilibrium points at undesirable values of 
n indicates the following potential problem with the operation of the system: for a long period 
of time the system may stay near the desirable stable equilibrium point, but due to occasional 
large stochastic fluctuations it may move to near the undesirable point. A drastic, and undesirable 
qualitative change in the performance of the stochastic system can occur rather suddenly. 

In many dynamical systems, a control mechanism based on feedback can be used to eliminate 
undesirable stable equilibrium points. That method is investigated for the ALOHA system in the 
next two sections. 
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4.2 Slotted ALOHA with Infinitely Many Stations 


No Control 


Let Az denote the number of new packets to arrive in slot k. Assume that the random variables 
(Az : k > 0) are independent, identically distributed with EJA] = A and Var(Az) < oo. For 
example, Az might have a Poisson distribution, which arises in the limit as the number of stations 
tends to infinity with the aggregate rate of traffic generation set to A. Suppose that each station 
with a packet at the beginning of a slot transmits in the slot with probability q. If there are n 
such stations then the probability of a success, Ps(n, qx), is given by Ps(n, qx) = nqu(1 — dr)"!. 
Since liMmn—oo Ps(n, qx) = 0, there is an no so large that Ps(n, qr) < A/2 for n > no. 

We shall argue that this implies that P|limķ—oo N(k) = oo] = 1. Conditioned on the event 
N(k) > no, the increment N(k + 1) — N(k) is stochastically greater than or equal to | a random 
variable u, where u = A; — B, and B is a random variable independent of A; such that P[B = 
1] = 4/2 = 1 — P[B = 0]. Note that Elu] = A/2 > 0. Moreover, given N(k) > no, the future 
increments of N are all stochastically larger than u until, if ever, the backlog becomes smaller 
than no. Furthermore, there is a positive probability that the backlog will never again be less than 
No, because of the following fact: If u1, u2, ... are independent, identically distributed random 
variables with E[u;] > 0, then P[mings; u1 + ... + ug < —1] < 1 (otherwise the Strong Law of 
Large Numbers would be violated). Thus, conditioned on the event N(k) > no, there is a positive 
probability that the backlog will converge to infinity and never again cross below no. Since the 
backlog must lie above no infinitely often, it follows that Pllimx-30 N(k) = +00] = 1. 


Centralized (Unrealistic) Control 


It is easy to show that for n fixed, Ps(n, qz) is maximized by qx = 1/n, that Ps(n,1/n) = (1 + 
1/(n —1))-@-) > el and that limp. Ps(1,1/n) = e~!. We can try to apply this result to a 
collision access system as follows. Let N(k) denote the number of stations with packets to transmit 
at the beginning of slot k, and suppose that each such station transmits its packet in the slot with 
probability 1/N(k). Then if N(k) is not zero, the conditional probability of a successful transmission 
in the slot given N(k) is greater than e~!. Thus the delay in the system is no larger than that for 
a discrete time M/M/1 bulk-arrival system in which the service times are geometrically distributed 
with parameter e~!. (The two processes can be constructed on the same probability space so that 
the M/M/1 bulk system always has at least as many packets in it as the ALOHA system.) The 
mean delay will thus be finite if A< e”! and the variance of the number of arrivals in each slot is 
finite. Unfortunately, this retransmission procedure is far from being practical because it requires 
that all the stations be aware of the number of stations with packets. 


Decentralized Control 


The stations can implement a practical variation of the centralized control scheme by estimating the 
number of stations with packets. Suppose, for example, that immediately after slot k the stations 


'“Stochastically greater than or equal to” refers to the notion of stochastic comparison, discussed in Section3.5. 
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learn the value of Z(k), where 


1 if there was a successful transmission in the slot 
e if there was a collision in the slot. 


O if the slot was idle 
Z(k) = 


The actual number of packets in the system evolves according to the equation 


where A(k) is the number of packets to arrive during slot k. By definition the term I{z(k)=1} is 
one if Z(k) = 1 and zero otherwise, and the stations learn the value of the term because they 
observe Z(k). On the other hand A(k) is independent of N(k) and all the feedback up to and 


including Z(k). Thus, a reasonable way for the stations to update an estimate N(k) of N(k) is by 
the following update equations: 


Ñ(k+1) = (Ñ (k) — Itza} + e(Z(k)))+ +A, (4.2) 


where c = (c(0),c(1),c(e)) is to be chosen so that the estimator error D(k) defined by D(k) = 
Ñ (k) — N(k) will tend to remain small for all k. Combining the equations for N(k + 1) and 
N(k+1), we see that if Ñ (k) > 1—min{c(0), c(1), c(e)}, then D(k+1) = D(k)—(A(k)—A)+c(Z(k)). 
Suppose that all stations with a packet to transmit at the beginning of slot k transmit in the slot 
with probability min(1,1/N(k)). The drift of (D(k)) during slot k is given by 


E[D(k + 1) — D(k)|N(k), Ñ(k)] = 
c(0)P[Z = 0|N(k), N(k)] +e 


+ c(1)P[Z = 1|N(k), N(k)] + c(e)P[Z = e|N(k), Ñ (k). 
It is not hard to show that if either N(k) or D(k) are sufficiently large then 


exp(—G(k)), 


3 
N 
I 

> 

= 
ce 
> 

D 
? 


where 


Therefore, the drift of D satisfies 
E[D(k +1) — D(k)|N(k), Ñ (k)] ~ d(G(k)), 
where 
d(G) = c(0) exp(—G) + c(1)Gexp(—G) + c(e)(1 — (1 + G) exp(—G)). 


We would like the sign of the drift of D(k) to be opposite the sign of D(k), in order that the error 
drift towards zero. Now D(k) is positive if and only if Ñ(k) is too large (i.e. larger than N(k)), 
which means that G(k) is less than one. Hence we want to choose the constants (c(0), c(1), c(e)) 
so that d(G) < 0 when G < 1. Similarly we want d(G) > 0 when G > 1. By continuity this forces 
d(1) = 1. Since we have only one equation (and two other conditions) for the three constants there 
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Figure 4.1: Drift field for ALOHA with distributed control. 


is some flexibility in choosing the constants. One suitable choice is c(0) = —(e—2)/(e—1) ~ —0.418, 
c(1) = 0 and c(e) = 1/(e — 1) ~ 0.582. To summarize, with this choice of (c(0), c(1), c(e)), if the 
stations use the update formula (4.2) and transmit with probability 1/N(k) in slot k when they 
have a packet, then the estimator error D(k) will tend to drift to zero. 

As a check on our calculations we can easily do an exact calculation of the drift vector field of 


the Markov process (N(t), N(t)), given by 


BR N(k +1) — N(k) N(k) \ (nm 
mae] Raro ra J Caw) CG )) 
This vector field is plotted in Figure 4.2 for A= 0.20 and the above value of c. 
The system should perform almost as well as the system with centralized control. The following 


proposition shows that the system with decentralized control is stable for the same arrival rates A 
as the system with centralized control. 


Proposition 4.2.1 Suppose (c(0),c(1),c(e)) are chosen so that d(G) < 0 if G < 1 and d(G) > 0 
if G > 1. Suppose E[Ax] = \ < e7}, and suppose for some constants a and b, that P[Az > x] < 
aexp(—bx) for all x > 0. Finally, suppose the system begins in the initial state N(0) = Ñ (0) = 0. 
Then there exist positive constants A and B with B < 1 so that, for all k, P[N(k) > n] < AB”. 


4.3 Bound Implied by Drift, and Proof of Proposition 4.2.1 


Proposition 4.2.1 will be proved in this section by a general method which we call the Lyapunov 
method of stochastic stability. The idea is to apply bounds implied by drift analysis (explained 
next) to a one-dimensional process which is a function of the main process under study. The 
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function mapping the original process into the one dimensional process is called the Lyapunov 
function. The method offers slick proofs, but, in typical applications, a good understanding of the 
original process and why it is stable, is needed in order to construct the Lyapunov function. 


Bounds Implied by Drift Analysis 


Let Yo, Yi,... be a sequence of random variables. The drift at instant k is defined by ElYx+1 — 
Yi Yo, Y1,- --, Yp]. Note that this drift is itself random, being a function of Yo,..., Yp. For notational 
convenience, we shall write Fẹ instead of Yo, Y¡,..., Yp when conditioning on these variables. Thus, 
the drift is E[Yx+1 — Yk| Fp]. If the sequence Yo, Y1,... is a Markov sequence, then the drift at instant 
k is a function of Y; alone. 

Let a be a constant with —0o0 < a < oo, and let €o, A and D be strictly positive finite constants. 
Consider the following conditions on the random sequence: 

Condition C.1 The drift at instant k is less than or equal to —e, whenever Y, > a, for k > 0. 
Equivalently, ? 


El(Ye+1 — Ye + €o)iyi>ap [Fr] $0 k20. 


One might expect that Condition C.1 alone is enough to guarantee that Y;, at least for large 
values of k, does not tend to be much larger than the constant a (if a is finite) or that Y, tends 
to —co in some sense as k — oo (if a = —00). However, an additional condition is needed (see 
homework exercise). It is sufficient that the increments of the sequence not be too large. That is 
the rough idea of Condition C.2: 

Condition C.2 There is a random variable Z with Elexp(AZ)] = D so that 


P||Yk+1 = lee Fee PZ > zt] cel 


Let c,7 and p be constants such that 


exp(AZ))= (14+ AE[Z) 
x 
UEFA 

n < €o/c and, 


p=1-— eon + cn’. 


c> El 


Then p < 1 and D > 1, and it is not difficult to show that Conditions C.1 and C.2 imply Condition 
D.1, stated as follows: 

Condition D.1 E[exp(n(Yk+ı — Yk) ty, > Fr] < p for k > 0. 

Also, Condition C.2 directly implies condition D.2, stated as follows: 

Condition D.2 Elexp(n(Yk+1 — a) iy, <a Fr] < D for k > 0. 


Proposition 4.3.1 Conditions D.1 and D.2 (and therefore also Conditions C.1 and C.2) imply 
that 


ok 
PIY: > HY] < p" exp 0) + TE Dexp(—n(b— a) (43) 


2The inequality should be understood to hold except possibly on a set of probability zero, because conditional 
expectations are only defined up to events of probability zero. 
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Note that the first term on the right-hand side of (4.3) tends to zero geometrically fast as 
k — oo, and the second term tends to zero exponentially fast as b — co. Note also in the special 
case Yo < a that (4.3) yields P[Y; > b|Yo] < pept), 

Proof. Note that Elexp(nYx+1)1Yo] = ElE[exp(nYk+1)|Fk]| Fo]. However 


Elexp(nYe+1)|Fx] = Eļexp(n(Yk+1 — Yk)) lty, >a} Fr] exp(nYk) 
+ Eļexp(n(Yk+ı — 0))L (7, <a}| Fr] exp(na) 
pexp(nYk) + D exp(na), 


IA 


so that 
Elexp(nYk+1)|Yo] < pElexp(7Yx)|Yo] + D exp(na). (4.4) 


Use of (4.4) and argument by induction on k yields that 


Elexp(nY.)|Yo] < p* exp(nYo) + 


Combining this inequality with the simple Chernoff bound, 
PLY, > b|Yo] = Plexp(n Y) 2 exp(nb)] < exp(—nb) Elexp(7Ye)|Yo], 


establishes (4.3). | 


Proof of Proposition 4.2.1 


In this section we explain how to prove Proposition 4.2.1 by applying Proposition 4.3.1. The drift 
of the backlog at instant k, (given the state of the system), is E[N(k + 1) — N(k)|N(k), N(k)]. 
If this drift were less than some fixed negative constant whenever N(k) is sufficiently large, then 
Proposition 4.3.1 could be directly applied to prove Proposition 4.2.1. By comparison with the 
centralized control scheme, we see that the drift is negative if the estimate N (k) is close enough 
to N(k). It is possible that N(k) is not close to N(k) (so that N(k) itself doesn’t have negative 
drift), but in that case the estimation error should tend to decrease. 


To turn this intuition into a proof, let Y, = V(N(k), N(k)) where V (n, à) = n+ (|fi—nl) with 


u? i 2 
da= R f0susk 
2ku-—kKk? ifu> K?. 


for some positive constant K to be specified later. Here V is the Lyapunov function which we use 
to convert the problem of establishing stability for the two dimensional process (N (k), Ñ (k)) into 
the problem of establishing stability for a one dimensional process. Thus, we argue that if K is 
sufficiently large, then the random sequence (Y;) satisfies Conditions C.1 and C.2 for some finite 
constants a, €o, A, and 7, and conditioning on Fẹ means conditioning on ((N(j), N(j)) :0 <j < k). 
Observe that Y, = N(k) + ¢(|N(k) — N(k)|) so that the drift of Y; is the sum of the drift of N(k) 
and the drift of ¢(|N(k)— N(k)|). To prove that Condition C.1 holds, note that if Y, is large, and if 
IN (k) — N(k)| is small, then Y; inherits negative drift from N(k), and the drift of 6(|N(k) — N(k)]) 
is small because the function ¢ is nearly flat near zero. On the other hand, if Yp is large, and if 
|Ñ (k) — N(k)| is also large, then Y, inherits large negative drift from $(|N(k) — N(k)|). Thus, in 
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either case, when Yp is large it has negative drift, so Condition C.1 is satisfied. Details are left to 
the reader. Condition C.2 is easily established—a key point is that the derivative of the function ¢ 
is bounded by 2K, so 


[Yesa — Ye| < (2K +1)|N(k +1) — N(k)| + 2K|N(k +1) — N(k)| (4.5) 


Thus, Proposition 4.3.1 can be applied to (Yp : k > 0), and since N(k) < Yp, Proposition 4.2.1 
follows immediately. 


4.4 Probing Algorithms for Multiple Access 


The decentralized control strategy described above has obvious deficiencies. For example, if a col- 
lision is observed, then until there is a success, it is clear that there are at least two backlogged 
stations. The update procedure would need a larger state space to take advantage of such infor- 
mation. One could imagine each station computing the conditional distribution of the number of 
backlogged stations, using a Bayesian update formula at each step (in fact this is described in 
the paper of A. Segall). This would be realizable if the state space is truncated to a finite set of 
states for the purpose of the estimator. Some approximate equations motivated by this procedure 
were given by Rivest. However, a key observation is that any given individual station knows not 
only the feedback information, but it also knows the times that it itself transmitted in the past. 
This information can serve to distinguish the station from other stations, thereby helping to re- 
solve conflicts. Algorithms called “probing algorithms” or “tree algorithms” take advantage of this 
additional information. 

First the basic collision resolution algorithm (CRA) is described. It operates in discrete time 
with the same 0—1—e feedback model considered for controlled ALOHA. Suppose there are initially 
k stations with packets to transmit. In the first slot, all stations transmit. If k=0 or k = 1, then 
at the end of the slot it is clear that all packets are successfully sent. If k=2 then a collision occurs 
in the first slot. 

After a collision, all transmitters involved flip a fair coin with possible outcomes 0 or 1. Those 
that obtain a 0 from their coin flip (those ” flipping 0” for short) retransmit in the very next slot, 
and those flipping 1 retransmit in the first slot after the algorithm is done processing the group 
of stations that flipped a 0. An example involving four stations with packets, A, B,C and D, is 
pictured in Figure 4.2. Suppose after the initial collision that stations B and C flip 0. Then both 
B and C transmit in the second slot, causing another collision. If then C flips 0 and B flips 1, then 
C and B are transmitted successfully in the third and fourth slots respectively. After the fourth 
slot, the stations with packets A and D realize it is time to transmit again, resulting in a collision 
in the fifth slot, and so on. 

The progress of the algorithm can be traced on a binary tree, as pictured in Figure 4.3. Each 
node of the tree corresponds to a single slot of the algorithm. Each terminal node of the tree 
constructed is labeled either 0 or 1, whereas each nonterminal node is labeled e. The algorithm 
corresponds to a virtual stack, as pictured in Figure 4.4. Packets all initially enter the top level, 
called level 0, of the stack. All packets at level 0 at the beginning of a slot are transmitted. If there 
is only one packet at level 0, it is thus successfully transmitted. If there are two or more packets 
at level 0, then all such packets flip a fair coin, and those that flip O stay at level O for the next 
slot, and all those that flip 1 move down to level 1. A packet that is in any other level l at the 
beginning of a slot is not transmitted in the slot. At the end of the slot the packet moves to level 


90 


1+ 1 if there is a collision in the slot, and moves to level | — 1 if there is no collision (i.e. either no 
transmission or a success) in the slot. For actual implementation of the algorithm, a station with 
a packet maintains a single counter to keep track of the level of the packet in the virtual stack. 

The basic CRA is now completely described. Some comments are in order. First, whenever there 
is a collision followed by an idle slot, the next slot will be a collision. For example, the outcomes of 
slots 5 and 6 in the example insure that slot 7 will be a collision. Consequently, the CRA can be 
“improved” by skipping a step in the algorithm. However, this improvement can lead to instability 
if the model assumptions are slightly violated—see problem XXX. Another observation is that all 
packets can sometimes be successfully transmitted a slot or more before the algorithm is complete. 
In the example, the algorithm requires eleven slots, even though all four packets are transmitted 
in ten slots. The eleventh slot is necessary in order for all stations listening to the channel outputs 
to learn that the algorithm is indeed complete. 

Let Ln denote the mean number of slots needed for a CRA to process a batch of n packets. 
Then Lo = Lı = 1. To find L, for n > 2, the outcome of the first slot is to be considered. Then, 


Ln 


1+ 2 HE E 2" Eladditional slots required|k stations flip 0 after first slot] 


= Elie "(Ly + En—1) 
= 14290 (paH 


Observing that Ln appears on the right side with a coefficient 2771), we can solve for Ln yielding 


the recursion: 
_ 142500 (p)? "Ek 
1 =91=1) 


Let 7 denote the (random) completion time given the number of packets has a Poisson distri- 
bution with mean u. The mean of 7 can be computed numerically using the expression 


W Ne 
2 pe Ln 
ID e a 


n=0 


(4.6) 


n 


A more direct way is to note that 


7 = El[number of nodes of tree visited] 


= 1+42£[number of nodes corresponding to collisions] 
0,0) 


= 1+2 Y 2* Pla given node in level k of full binary tree corresponds to a collision] 
k=0 
OO 

= 142) 2{1 — (1+ p2™) exp(—27*)) 
k=0 


The final sum converges quickly, as can be seen from the fact that 1—(1+¢) exp(—e) = e?/2+o(e?) 
as € — 0. Thus, 7 is finite for any u > 0. 
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4.4.1 Random Access for Streams of Arrivals 


So far a method for arranging for transmissions of a fixed group of stations with packets has 
been described. This collision resolution algorithm can be used as a building block to construct a 
random access strategy for a continuous stream of arrivals of new packets. In one method, packets 
just arriving are first transmitted in the first slot after the batch in progress has been handled. This 
leads to large variations in batch size. Analysis can be based on the observation that the sequence 
of consecutive batch sizes forms a Markov process. A second method, known as continuous entry, is 
discussed in Problem xxx. A third method, known as the decoupled window strategy, is discussed 
in the next subsection. 


4.4.2 Delay Analysis of Decoupled Window Random Access Scheme 


Under the decoupled window random access scheme, packets are grouped into batches. The kt” 
batch consists of those packets that arrive during the time interval [(k — 1)A,kA), where A is 
a parameter of the algorithm. Note that the distribution of the number of packets in a batch is 
Poisson with mean AA. Packets in each batch are processed by a separate run of the basic collision 
resolution algorithm. The processing time of the kt” batch is denoted 7;. The variables 7,,k > 0 
are independent and each has the same distribution as the completion time 7 of the basic CRA 
when the number of packets involved has the Poisson distribution with mean y = AA. Processing 
of the kth batch begins in the first slot after the (k — 1)*” batch is processed, or in the first slot 
after time kA, whichever is later. 

Let U; denote the waiting time of the kth batch, which is the amount of time from kA until 
processing for batch k begins. Note that for suitable initial conditions, Up = Up + yx, where yz is 
uniform on the interval [0,1], and Uf is the waiting time of the kth batch in a modified system which 
does not have time slotting of the transmission process (cf. section on servers with vacations). The 
sequence (Uf : k > 1) satisfies the equations: 


Ups, = (Uk — A+ Tk)+, 


which is identical to the update equations for the waiting times in a GI/GI/1 queue (more specif- 
ically, a D/GI/1 queue), if the 7 represent the service times in the GI/GI/1 queue, and if the 
interarrival times are the constant A. For E[U;] to be bounded as k — oo it is necessary and 
sufficient that 7 and Var(r) be finite. In fact, by Kingman’s first moment bound, 


AVar(9) + Var(A))  AVar(r) 


E[UZ] < X1- F/A) -2(1—-7/A) 


if the load p = F/A is less than one. It can be shown that r has finite variance for any A > 0 by 
modifying either of the two methods used to compute 7. 

For E[Uf] to be bounded it is thus sufficient that A < AA/T = u/T, where u = AA. For the 
basic CRA, the maximum throughput is max, 4/7 = 0.429. ZZZ CHOICE OF A. It is 0.462 for 
the modified CRA. 

The maximum throughput can be further increased by terminating the processing of a batch 
whenever the conditional distribution of the number of remaining packets is Poisson distributed. 
Those packets can be regrouped with packets arriving during a different time interval with length 
adjusted so that the mean number of packets in the resulting batch is adjusted to an optimal value. 
This leads to maximum throughput 0.4872. Finally, with this restart idea used, the tree is no longer 
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symmetric, so further improvement can be gained by adjusting the bias of the coin used, yielding 
maximum throughput 0.4878. 


4.5 Problems 


(I haven't assigned these problems recently, and solutions are not provided for them in the back. 
-BH) 

4.1. M/G/1 queue with bulk arrivals 

Consider a single server queueing system in which customers arrive in batches. Suppose that batches 
arrive according to a Poisson process of rate A, and suppose that Pla batch has k customers]=gz, 
independently of the time of arrival and of the size of all other batches. Suppose the service times 
of all customers are independent with distribution function B(x) and mean 1/p, and suppose the 
sequence of service times is independent of the arrival process. Suppose that the customers are 
served in FCFS order, and that the customers within a batch are served in some random order that 
is independent of the service times of the customers. Think of a “typical” customer as one drawn at 
random from among the first N customers, all choices being equally likely, where N is very large.” 
// (a) Define the waiting time of a batch to be the length of time from when the batch arrives until 
some customer in the batch begins service. Under what condition is the limiting batch waiting 
time finite? Under such condition, what is the mean waiting time of a batch in equilibrium? (Hint: 
reduce to consideration of an M/G/1 system). 

(b) What is the distribution of the total number of customers in the same batch as a typical 
customer? Given that a typical customer is in a batch of size n, what is the conditional distribution 
of the number, X, of customers in the same batch as the typical customer which are served before 
the customer itself is served? What is the unconditional probability distribution of X? What is 
E[X]? Double check your answers for the case gı = g2 = 0.5. 

(c) What is the mean waiting time in queue for a typical customer? 

(d) (Effect of statistical multiplexing). If the arrival rate is doubled to 2A, if the batch sizes have 
the same distribution, and if the service times are cut in half, how does the mean waiting time in 
the queue change? How about the mean waiting time in the system? 

(e) (Effect of increasing burstiness). If the arrival rate is halved to A/2, if the batch sizes are 
doubled, and if the service times are kept the same, how does the mean waiting time in the queue 
change? 

4.2. Basic ALOHA 

Consider a basic ALOHA system with M stations, each having buffer size one, as discussed in 
class. (a) Suppose the arrival probability per slot at an empty station is qa = 1/(Me), and that the 
transmission probability per slot at an occupied station is q, = 1/M. Perform a drift analysis of 
the system. How many equilibrium points are there? Assuming M is large and approximating the 
mean number of busy stations using drift analysis, derive the throughput and the mean delay. (b) 
Continue to assume that M is large. Also assume that q, = 3/M and qa = a/M for some constant 
a. For what value of a is T ~ M/2 (in other words, about the same as for TDMA), where the mean 
delay T is again predicted by drift analysis. How many equilibrium points are there for this value 
of a? 

4.3. Basic ALOHA for finite M 

Consider a slotted ALOHA system with M stations, each with an infinite buffer. Each station 
which, at the beginning of a slot, has at least one packet, transmits a packet with probability 
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qz. In addition, each station, whether transmitting or not, receives a packet during a slot with 
probability qa. Suppose the stations have unbounded buffers. At the end of each slot, all stations 
learn how many packets were transmitted. If exactly one is transmitted, that packet leaves the 
system. Otherwise, no packets leave the system. (a) Identify a Markov chain describing the system 
and give the one-step transition probabilities. (Hints: What is the state space? To gain intuition, 
it might help to think about the case M = 2 first.) (b) What is a necessary and sufficient condition 
for the system to be stable? (Be as explicit as possible. You should explain your reasoning but you 
needn’t prove your answer.) 


4.4. Drift analysis for two interacting Markov queues 

Consider a system with two queueing stations. Suppose the arrival processes to the two stations 
are independent Poisson processes with the rates A; and A2. Suppose the stations are correlated 
in the following sense. When there are nı customers in the first station and ng customers in the 
second station, than the instantaneous service rate in the first station is yni/(nj + n2) and the 
instantaneous service rate in the second station is yn2/(ni + n2). That is, the effort of a rate u 
exponential server is allocated among the two stations in proportion to their numbers of customers. 
a) Sketch the drift vector field of the Markov chain (Nj, N2) assuming that Ay = 1,2 = 2 and 
u = 4. b) Find the mean total number in the system in equilibrium. c) Under what conditions on 
A1, A2 and p is the Markov chain positive recurrent? Can you prove your answer? 


4.5. Is negative drift alone enough? 

Let X1, X2,... be independent random variables with P[X; = —2k?] = 1/k? and P[X; = 1] = 
1 — 1/k?. Let Yy = Xı + X2 +... + Xk. (a) Find di (n) = ElYk+1 = Ye] Yi, co YESS YS n]. What 
does your answer suggest about the asymptotic behavior of the sequence (Yp : k > 1)? (b) Let 
Ap be the event {Xp = —2k?}. Since the sum P[A1] + P[A2] +... is finite, the Borel-Cantelli 
lemma implies that P[A; occurs for infinitely many k] = 0. What does this fact imply about the 
asymptotic behavior of the random sequence (Y, : k > 1)? (Does the discrepancy between your 
answers to (a) and (b) contradict any results from class?) 


4.6. Controlled ALOHA 

Consider a discrete-time controlled ALOHA system. Suppose that just before slot one there are 
no packets in the system. The number of new packets that arrive during each slot has the Poisson 
distribution with mean A, and each new packet arrives at an empty station (infinite user model). 
The numbers of arrivals during distinct slots are mutually independent. Each station keeps track 
of a counter. We use C(k) to denote the (common) value of the counters at the beginning of slot 
k. We assume C(1) = 0 and use the following update rule: C(k) = (C(k) — 1), if no packets are 
transmitted in slot k, and C(k) = C(k) +1 if at least one packet is transmitted in slot k. Suppose 
that any station with a packet at the beginning of slot k transmits it with probability f(k), where 
f(k) = 1/C(k) if C(k) > 1 and f(k) = 1 otherwise.(a) Identify a Markov chain describing the 
system and write down the transition probabilities. (b) For what range of is the Markov chain 
positive recurrent? (Hint: When the backlog is large, what does the mean number of packets 
transmitted per slot drift towards?) 


4.7. A controlled ALOHA system 

Consider a discrete-time controlled ALOHA system, as discussed in class. Suppose that just before 
slot one there are no packets in the system. The number of new packets that arrive during each 
slot has the Poisson distribution with mean A, and each new packet arrives at an empty station 
(infinite user model). The numbers of arrivals during distinct slots are mutually independent. Each 


94 


station keeps track of an estimate N(k) as follows: 
Ñ(k +1) = (N(k) — Liz(1)1) + (Ze) 4 + e7 


where (c(0),c(1),c(e)) = (—0.418,0,.582). (If the e”! here were replaced by A, this would be 
identical to the estimator considered in class. The estimator here does not require knowledge of 
A.) Suppose that any station with a packet at the beginning of slot k transmits it with probability 
min(1/N(k),1). (a) Identify a Markov chain describing the system, and write down the transition 
probabilities. (b) When N(k) and N(k) are both large, the difference Go Ñ (k)— N (k) drifts towards 
zero for some value of Go. How is the constant Go determined? (Be as explicit as possible.) (c) 
Suppose for a given A that you have computed Go. Under what condition will the system be stable 
(using any reasonable definition of stability)? (d) Extra credit: For what range of A is the system 
stable? 

4.8. An access problem with three stations 

Three stations each have a packet and contend for access over a time-slotted channel. Once a station 
successfully transmits its packet, it remains silent afterwards. Consider two access methods: (i) 
Each station transmits in each slot with probability 1/2 until successful. (ii) The stations use 
the basic unimproved tree algorithm based on fair coin flips. (a) For each of these two methods, 
compute the expected number of slots required until all packets are successfully transmitted. (b) 
Suppose a fourth station is monitoring the channel, and that the fourth station does not know that 
there are initially three stations with packets. Compute the expected number of slots until the 
fourth station knows that all packets have been successfully transmitted. 

4.9. Basic tree algorithm with continuous entry 

Packets are transmitted in slots of unit length. Suppose packets arrive according to a Poisson 
process of rate À packets per slot. Each packet arrives at a distinct station, so that any station has 
at most one packet. Each station adopts the following strategy: 

* A counter value C is set to zero when a new packet is received. 


* Tf a packet has counter value Č at the beginning of a slot, then 
if C = 0 the packet is transmitted in the slot. If the transmis- 


sion is successful the packet leaves the system. If a collision 
occurs, the station flips a fair coin (value zero or one) and 


sets C equal to the value. 
else if C > 1 the packet is not transmitted in the slot. If no 


—collision occurs in the slot then C is decreased by one. If a (a) Sketch the movement of packets 


collision occurs then C is increased by one. 
in a virtual stack. (b) Suppose n new packets arrive just before slot zero and that there are no 


other packets in the network at that time. Let B, denote the expected number of slots until the 
feedback indicates that there are no packets in the network. Then Bp = Bı = 1. By conditioning 
on the outcome during the first slot, find equations for By, B3,... Can you find the least upper 
bound on the set of A such that Bn is finite? 

4.10. Simple multiple access with four stations 

Four stations, labeled 00,01,10,11, share a channel. Some subset of the stations wish to transmit 
a packet. The stations execute the basic tree algorithm using the bits in their labels to make 
decisions about when to transmit. Let R; denote the mean number of slots needed for this conflict 
resolution algorithm to complete, given that k of the four stations (chosen at random, uniformly) 
have a packet to transmit. Include the first slot in your count, so that Ro = Rı = 1. (a) Find 
Ra, R3 and R4. (b) Compare to the mean time if instead the decisions in the basic tree algorithm 


95 


are made on the basis of independent fair coin flips for 1 < k < 4. Also consider the comparison 
for small values of k when there are 27 stations for J equal to 5 or 6. 
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Figure 4.2: Time axis for the basic collision resolution algorithm 


Figure 4.3: Tree for the basic collision resolution algorithm 
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Figure 4.4: Dynamics of the counters for the basic collision resolution algorithm 
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Chapter 5 


Stochastic Network Models 


This chapter describes simple stochastic models for two types of networks: circuit switched networks 
and datagram networks. In circuit switched networks, a given route is considered to be available if 
there is sufficient capacity at each link along the route to support one more circuit. A performance 
measure of interest for a circuit switched network is the probability a given route is available. In 
datagram networks, individual customers move along routes in the network. Performance measures 
of interest for datagram networks are the transit delay and throughput. 

Although arbitrary networks can be quite complicated, the models considered in this chapter 
have surprisingly simple equilibrium distributions. Furthermore, they suggest useful approxima- 
tions (such as the reduced load approximation for blocking probability) and they illustrate impor- 
tant phenomena (such as the importance of bottleneck links in heavily loaded store and forward 
networks). 

Before the two types of network models are presented, the theory of reversible Markov processes 
is introduced. Among other things, the theory of reversible Markov processes suggests rather general 
techniques for constructing complicated stochastic models which have fairly simple equilibrium 
distributions. 


5.1 Time Reversal of Markov Processes 


The reverse of a random process X = (X(t): t € R) relative to a fixed time 7 is (X(T —t):t €R). 
Recall that (1.5) gives a characterization of the Markov property which is symmetric in time. 
Therefore a random process (X(t) : t € R) is Markov if and only if its reverse is Markov. If X is 
Markov, the transition probabilities of its reverse process are obtained by 


P[X(t) = JPIX(t +s) = jX = 4 


P[X(t) = i|X(t +s) = j| = P[X(t+s) = j] 


Note that even if X is time-homogeneous, the reverse is not necessarily time-homogeneous. 
Suppose (X(t) : t € R) is a pure-jump, stationary Markov process with countable state space S 

and irreducible generator matrix Q. Then the reverse process is also stationary, and its equilibrium 

distribution 7”, transition probabilities (p};(t)), and generator matrix Q’ are given by 

O: SG. ad q = Hi, (5.1) 

Tj Tj 


T =T, Pjilt) z 
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In vector-matrix form, this last relation becomes 
Q! = diag(r) *Q* diag(r) 


where 
diag(t) =| 9 72 


For example, the transition rate diagram of a stationary Markov process is shown in Fig. 5.1, along 
with the transition rate diagram of the corresponding reverse process. Note that the process and 
its reverse have different transition rates for any a > 0. However, the sum of the rates leading out 
of any given state is the same for the forward and reverse time processes. This reflects the fact 
that the mean holding time for any given state, which is the reciprocal of the sum of the rates out 
of the state, should be the same for the forward and reverse processes. 


Figure 5.1: Example of forward and reverse transition rate diagrams 


The notion of reverse process is useful for verifying that a given distribution is the equilibrium 
distribution, as indicated in the following proposition. The proof is straightforward and is left to 
the reader. 


Proposition 5.1.1 Let X be an irreducible, pure-jump Markov process with generator matrix Q, 
let m be a probability vector and Q’ be a generator matrix such that 


Tiqij = tjl; Vij ES. 
Then n is the equilibrium distribution of X and Q' is the generator of the reverse process. 


A random process is reversible if the process and its reverse (relative to a fixed time 7) have 
the same finite-dimensional distributions, for all 7. Due to the phrase “for all 7” in this definition, 
a reversible process is necessarily stationary. 

The pure-jump stationary Markov process X is reversible if and only if Q = Q’, or equivalently, 
if and only if the detailed balance equations hold: 


Tiqij = Tjqji Vi, j. (5.2) 
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In comparison, the global balance equations are TQ = 0, or equivalently 


y Ti ij = So TQ ji Vj. 


iF inAj 


Note that it is easy to construct reversible Markov processes as follows. First select a state space 
S and an arbitrary probability distribution m on S. Then for each pair of states ¿ and j, simply 
select any values for qj; and qj; such that (5.2) holds. Also, if S is infinite, care should be taken 
that the sum of the rates out of any state is finite. 

Reversible Markov processes also arise naturally. Consider for example an irreducible, stationary 
birth-death processes, and let n > 0. The only way the process can move between the subsets 
{0,...,.n} and {n+1,n+2,...} is to make a jump from nto n+1 or from n+1 to n. Thus, TrAny1 = 
Tnt+ifn+1- This relation yields that the detailed balance equations (5.2) hold, so that the process 
is reversible. 

More generally, let X be an irreducible stationary pure-jump Markov process, and let G = (V, E) 
denote the undirected graph with set of vertices S and set of edges E = {[i, j] : i # j,and qij > 
0 or qji > 0). The graph G is called a tree graph if it is connected (which it must be because X 
is assumed to be irreducible) and cycle-free. In case S is finite this is equivalent to assuming that 
G is connected and has |S| — 1 edges, or equivalently, G is connected but the deletion of any edge 
from G divides G into two disjoint subgraphs. The same reasoning used for birth-death processes 
(which are special cases) shows that any stationary Markov process so associated with a tree graph 
is reversible. 

A useful way to obtain reversible processes with larger state spaces is to consider collections of 
independent reversible processes. That is, if X(t) = (X1(t),...,Xn(t)) for all t, where the compo- 
nents X;(t) are mutually independent reversible processes, then X itself is a reversible process. 

If X is an irreducible reversible pure-jump Markov process with generator matrix Q, then a new 
such process can be obtained as follows. Fix any two distinct states 1, and jo and let a > 0. The 
new generator matrix is the same as Q, except that q,;, and q;,i, are replaced by aq;,;, and aqj,i, 
respectively, and the new values of q;,;, and q;,;, are chosen so that the new matrix also has row 
sums equal to zero. We can as well take a = 0, although doing so might destroy the irreducibility 
of the resulting Q matrix, so that more than one modified process can arise, depending on the 
equilibrium distribution. 

Taking this last idea a bit further, given a subset A of S we can obtain a generator for a Markov 
process with state space A by simply restricting Q to the set A x A, and suitably readjusting the 
diagonal of Q. The resulting process is called the truncation of X to A. 

As we shall see in the next section, the above ideas of considering collections of processes (to 
build up size) and truncation (to introduce nontrivial dependence among the constituent processes) 
is a powerful way to construct complex reversible models in a natural way. 

Exercise: Consider the above modification if a — oo. In some sense the states i, and jọ become 
aggregated in this limit. 


5.2 Circuit Switched Networks 
Consider a circuit switched network with R routes and M links. A route is simply a subset of links. 
Usually such links form a path, but that is not necessary in what follows. An example network with 


links {1,2,3,4} and four routes {1,2,3},{1},{2,3},{2,4}, is shown in Figure 1.2. Suppose that calls 
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arrive for route r according to a Poisson process at rate vp, and that call durations are independent 
and exponentially distributed with mean one. 

Let X,(¢) denote the number of calls in progress on route r at time t, and let Z(t) denote the 
resulting load on link l, defined by 


Z(t) = Y Ar X(t), 


where A is the link-route incidence matrix defined by A), = 1 if l € r and Aj, = 0 otherwise. In 
matrix notation, Z(t) = AX(t) for all t. 

Then X = ((X1(t),..., Xp(t))" : t > 0) is a Markov process with state space ZR. If there 
are no capacity constraints imposed on the links, then the components of X are independent, and 
each behaves as an M/M/oo system. An M/M/oo system is of course a special type of birth-death 
process, and is thus reversible. Therefore X itself is reversible. 


Figure 5.2: A sample circuit switched network, with routes indicated. 


Let e; denote the column vector with a 1 in the ith position and 0’s in all other positions. The 
Q matrix is given by (only nonzero, off-diagonal entries are displayed): 


q(2,2+e,) =1% ifr EZR 
q(x,2—er) =a, ifs—e, €Z 


and the equilibrium distribution 7 is given by 


n(x) = II v;"exp(—vr) 


> Ly! 
We remark that the process Z is not necessarily Markov. Consider the example in Figure 5.2. If it 
is known that Z(t) = (1,1,1,0), then it is not determined whether there is a single call, on route 1, 
or two calls, one on route 2 and one on route 3. The history of Z before t can distinguish between 
these two possibilities, giving additional information about the future of Z. 
If capacity constraints are imposed, so that link l can support at most Cı calls, then the state 
space is truncated down to Sc = {x € ZR : Ax < C}, where C is the column vector of capacities. 
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The truncated system is also reversible, and the equilibrium distribution! is 


cy 1 Ul v2" exp(—v;,-) 


Ly! 


where the constant Z(C) is chosen to make © a probability distribution. 

In the remainder of this section we examine the probability of blocking for calls offered for a 
given route. If the network has only a single link, the total number of calls in progress behaves like 
an M/M/C1/C1 system, so that 


k 
P[k calls in progress] = A 
ys j=0 dy q 
where A, is the sum of the arrival rates. In particular, the blocking probability at the link is given 
by Bı = P[C¡ calls in progress] = E(A1,C1), where E denotes the Erlang blocking formula. 

In general, a link / is full with probability Bı = 1 — (Z(C — e;)/Z(C)). For large networks, this 
is a difficult quantity to compute, which leads us to consider bounds and approximations. First, a 
bound is presented. Define A, for a link l as the sum of vp, over all routes r that include link l. In 
matrix notation, this is A= Av. The following conservative bound on the link blocking probability 
is intuitively clear: 


Bı < E(A¡, Ci) (5.3) 


A proof of (5.3) goes as follows. Fix a link J. The right hand side, E(A;,C;), is the same as the 
blocking probability in the original network if all link capacities except Cı are changed to +00. It 
is easy to construct the original network process and the modified process on the same probability 
space so that, with probability one, the number of calls in progress through line / for the original 
process is less than or equal to the number of calls in progress for the modified process. Indeed, 
the sequences of offered arrivals should be the same, and departures of routes currently using link 
l should be governed by Cı independent Poisson clocks. When there are k > 1 customers in a 
system (either one) at time t— then there is a departure at time t if one of the first k Poisson clocks 
rings. In particular, the fraction of time link / is full is smaller for the original process. Since time- 
averages equal statistical averages for positive recurrent Markov processes, (5.3) follows. What is 
less obvious (and more difficult to prove) is the remarkable lower bound of W. Whitt (“Blocking 
When Service is Required from Several Service Facilities,” AT&T Technical Journal, vol. 64, pp. 
1807-1856, 1985) 

Plroute r free] > [[a — E(A1, C1)) (5.4) 


ler 


The bounds (5.3) and (5.4) are often accurate approximations when blocking probabilities are 
very small (which is, of course, often the case of interest). For moderately loaded networks, the 
following reduced load approximation is often quite accurate: 


By, = E(Á, Ci), (5.5) 


‘Amazingly enough, the same equilibrium distribution results if the holding times of calls for each route r are 
independent and identically distributed (not necessarily exponentially distributed) random variables of unit mean. 
This is an example of an “invariance” result in queueing theory. 
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where Ni is an estimate of the effective traffic offered at link l: 


M= > 4% [[ G- By) (5.6) 


riler Vleer lAl 


The product term in (5.6) approximately accounts for the thinning of the traffic offered at link | 
due to blocking at other links. Taken together, (5.5) and (5.6) yield 2M equations for the 2M 
variables (É; A : 1 < 1 < M). Moreover, the \’s are determined by the B’s, and the B's are 
determined by the Ys. The equations can thus be thought of as fixed point equation for the B’s 
(or for the \’s). They can be numerically solved using iterated substitution or dampened iterated 
substitution. It can be shown that there is a unique fixed point (F.P. Kelly, “Blocking Probabilities 
in Large Circuit Switched Networks,” Advances in Applied Probability, 18, 473-505, 1986). The 
acceptance probability for a route r can then be approximated as 


Plroute r free] = II (1- Ê). (5.7) 
ller 


Theory and numerical experience have shown the fixed point equations to be rather accurate for 
large networks with large capacity constraints and a diverse collection of routes. 


5.3 Markov Queueing Networks (in equilibrium) 


Networks of queueing stations (we use the word “station” instead of “system” here to better distin- 
guish the elements from the total network) are examined in this section. A simple way to construct 
such a network is to arrange a sequence of stations in series. Such networks are considered first. 
Then two more general types of networks are considered, which differ in the routing mechanisms 
used. In the first, each customer wanders through the network according to a Markov process. In 
the second model, each customer follows a deterministic route, which depends on the type of the 
customer. If one views the type of a new customer as being random, one can see that the second 
routing mechanism is essentially more general than the first. For simplicity, we restrict our attention 
to Poisson arrival processes and Markov type service mechanisms. More general networks, based 
on the notion of quasi-reversibility discussed below, are described in Walrand, An Introduction to 
Queueing Networks, Prentice Hall, 1988. 


5.3.1 Markov server stations in series 


We first consider a single station, paying particular attention to the departure process. A “Markov 
server” with state dependent rate, denoted as an Mg type server, with parameters (1), u(2),..., 
is an exponential server with instantaneous rate u(n) when there are n customers in the station. 
That is, given there are n customers in the station at time t, the probability of a service completion 
during the interval [t,t + h] is u(n)h + o(h). If the arrival process is Poisson with rate A, the state 
of the resulting number-in-station process is a birth-death Markov process of the type considered 
in Section 1.12, such as an M/M/m station with 1 < m < oo. The equilibrium distribution exists 
and is given by: 


An 
———~, if = +00. 
51 h= 40) = > T in p(l 


t(n) = 
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Suppose the state process (N(t)) is stationary. Since it is a birth-death process it is time-reversible. 
Conclude that the following properties hold. The first property, which implies the second, is known 
as quasi-reversibility. 


e The departure process before time t is independent of N(t). 
e The departure process is Poisson, with rate A. 


Next, consider a network of Mg stations in series, with the arrival process at the first station 
being a Poisson process with rate A. The entire network is described by a Markov process with 
state space S = [(n1,n2,...,ny):n € Z4} = Z]. Consider a stationary version of the network 
state process (which exists if the station is positive recurrent.) Also, fix a time to. Since there is no 
feedback effect in the network, the first station in the network behaves like an isolated Mg server 
station with Poisson arrivals. Therefore, the departure process from the first station is a Poisson 
process. In turn, that means that the second station in the network (and by induction, all other 
stations in the network) behave as an isolated Mgs server station with Poisson arrivals of rate A. 

Of course the individual stations are stochastically coupled, because, for example, a departure 
instant for the first station is also an arrival instant for the second station. However, remarkably 
enough, for a fixed time to, the components Nj(t,),...,N s(t.) are independent. One explanation 
for this is the following. By the first quasi-reversibility property for the first station, Ni(to) is 
independent of departures from station 1 up to time ty. But those arrivals, together with the 
service times at the other stations, determine the states of all the other stations. Therefore, Nj (to) 
is independent of (Na(to),..., Ny(to)). Similarly, N2(to) is independent of the past departures from 
the second station, so that Na(t,) is independent of (N3(to),..., Ny(to)). Continuing by induction 
yields that Ni(t.),...,Ns(t.) are indeed independent. 

Therefore, the equilibrium distribution for the network is given by 


J 


r(n1,n2,...,n)= pj(n;) where pj(nj) = =z; —~ 
I ici ota I (D 


where b; is a normalizing constant for each j. 

Another approach for proving this result is to check that TQ = 0. That approach is taken in a 
more general setting in the next subsection. An advantage of the argument just given is that it can 
be modified to show that the waiting times of a typical customer at the J stages of the network are 
mutually independent, with each waiting time having the distribution of a single Mg server station 
with Poisson arrivals in equilibrium. 


5.3.2 Simple networks of Ms stations 


Consider next a network of J stations, each with Mg type servers. Let u;(n) denote the service rate 
at station j given n customers are at the station. Customers are assumed to follow routes according 
to a discrete time Markov process with one step transition probability matrix R = (rij)i<ij<J- 
Thus, given that a customer just left station i, it visits station j next with probability rij. We 
distinguish two types of networks, open and closed, as follows. 

An open network accepts customers from outside the network. Such arrivals are called erogenous 
arrivals. The stream of exogenous arrivals at any given station į is assumed to be a Poisson stream 
with rate 7;, and the arrival processes for distinct stations are assumed to be independent. To 
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avoid an infinite buildup of customers, we assume that R is such that starting from any station in 
the network, it is possible to eventually leave. In particular, at least some of the row sums of the 


matrix R are less than one. The vector of exogenous arrival rates y = (y1,..., YJ) and R determine 
the vector of total arrival rates A = (A1,..., AJ) by the equations 
Ài = Yi + y AGT ji Vi, (5.8) 


J 


which in vector-matrix notation can be written À = AR + 7. 

A closed network contains a fixed set of customers. There are no exogenous arrivals, nor 
departures from the network. The rows of R all have sum one. A possibly unnormalized vector of 
arrival rates A = (\1,..., AJ) satisfies 


Aj = F AS Vi (5.9) 
J 


or in vector form, A= AR. These are the same equations as satisfied by the equilibrium distribution 
for a finite state Markov process with one step transition probability matrix R. Assuming that R 
is irreducible (as we do), the vector A is uniquely determined by these equations up to a constant 
multiple. 

If for each j the service rate ju;(n) is the same for all n > 1 (so that the service times at station 
j are exponentially distributed with some parameter uj), then the network model of this section is 
known as a Jackson network. 


Proposition 5.3.1 The equilibrium distribution of the network is given by 


J 
r(n1,n2,...,n7) = G7! [ [vino 
i=1 
where 
Ay 
Ne TE 
and 
J 
6=¥ (loo) 
n \i=l 
where the sum is over all possible states n = (n1,... ng). (For closed networks, write G(K) instead 


of G to emphasize the dependence of G on K). 


Proof. In order to treat both open and closed networks at the same time, if the network is open 
we convert it into a closed network by adding a new station which represents the “outside” of 
the original open network. That is, if the network is open, add a station called station 0 (zero) 
which always has infinitely many customers (so ny = +00). Let the service rate of station 0 be 
Lo(no) = Ytot = ys yi and extend the routing matrix by setting 


J 


To = Ni 5 roo = 0, and rio = 1- S Ry 
Vtot 


j=l 
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The extended routing matrix R has row sums equal to zero, and the vector A satisfying (5.8) can 
be extended to a vector satisfying (5.9) for the extended network by setting Ao = Jtot. Then the 
network can be considered to be closed (albeit with infinitely many customers) so the remainder of 
the proof is given only for closed networks. 

For state n = (n1,ng....,nz) let Tijn denote the new state 


Tijn = fio ig 1,... nj +1,... nJ) OSij <J. 


The transition rate matrix for the Markov process describing the whole network is defined by (only 
terms for n Æ n’ need be given): 


Quans pilni)rig ifn’ = Tijn some ij 143 
i 0 ifn’ A Tijn for any ij and n£n 


We need to verify the balance equations 


D m(n')Q(n',n) = 5 m(n)Q(n,n’) for all n. 


n'n! En ninán! 


Therefore it suffices to note that the stronger statement holds for each ¿ such that n; > 1: 
y TT ¡¡n)Q(T,¡n,n) = > T(n)Q(n, Tijn), Vn, Vi 
sii jjži 

which is equivalent to 


2 pw Gory Gta) | ee ele Mines 


FIA 


which is true because it reduces to (5.8) with y; = 0. a 


TO BE ADDED NOTES HERE: Evaluation of mean delay for open networks, and evaluation 
of throughput/cycle time for closed networks and investigation for varying number of customers, 
application to head of line blocking in packet switches. 


5.3.3 A multitype network of My stations with more general routing 


We assume the network has / types of customers. It is assumed that new type ¿ customers are 
generated according to a Poisson process with rate v(i) and that each type i customer has a fixed 
route, equal to the sequence of stations (r(i, 1), r(i,2),...,r(i, s(1))). 

We assume that uj(nj) is the departure rate of station j if there are n; customers there, no 
matter what type of customers are in the station, for (1 < j < J). For simplicity, we assume that 
customers at each station are served in first-come, first-served (FCFS) order. (Results exist for 
many other orderings, but the service order does play a big role in the model.) 

The entire network can be modeled as a pure jump Markov process. To obtain a sufficiently 
large state to obtain the Markov property, it is necessary to keep track not only of the number of 
customers n; at each station j, but also the type t;(1) of the It? customer in station j for 0 < j < nj. 
Moreover, since we allow a customer to visit a given station more than once, in order to know the 
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future routing we must also keep track of s;(/), which is the stage that the [th customer in station 
j is along its route. For example, if sj(l) = 3 then the lth customer in station j is currently at the 
third station (which happens to be station j) along its route. A generic state of the network can 
thus be written as c = (c1,C2,..., CJ), where for each j, ej is the state of station j, written as 


Cj = (c;(1), c;(2),. 2 SEAT) 


where c; (I) = (t;(1), s;(1)) is called the class of the I** customer in station j. 
The long term total arrival rate at station j (assuming the network is positive recurrent) is 
given by 
= y v(i) x { number of times route ¿ passes through station j} 
types i 


Define a probability distribution for the state of station 7 by 


cg E TO) 
| n z I 7 ll 20 l 


=1 


Under this distribution, the distribution of the total number in station j is the same as in the 
simpler network of the previous subsection. Furthermore, given nj, the classes attached to the 
customers in the n; positions in station j are independent and proportional to the fraction of 
arrivals of various classes. 


Proposition 5.3.2 The equilibrium distribution for the network is 


Proof. Given a state c of the network, let T’c (respectively T;c) denote the new state caused 
by an arrival of an exogenous customer of type i (respectively caused be a departure at station j, 
which is well defined only if n; > 1). Then, the transition rates for the network (given only for 
c# Cc’) are 


pim if œ Se 


Oe,c)=<¢ v(i) if d=Tie 
0 if œ is not of the form T'e or Tje (and cx e). 


Now, some thought (or hindsight!) allows us to make the following guess for transition rates of 
time-reversed network: 

v(i) if œ = Tje and s;(1) = S(t;(1)), where i = t;(1) 
Q'(cd',c)= <4 ukl(ng+1) if œ= Tjc and s;(1) < S(t;(1)), where k = r(t;(1),s;(1)+ 1) 

vn +1) if œ= Tic where k = r(j,1). 


To complete the proof we need only verify that 7(c)Q(c, e”) = r(c)Q'(e, c) for all states c, e” (this 
is left to the reader) and apply Proposition 5.1.1. E 
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5.4 Problems 


5.1. Time reversal of a simple continuous time Markov process 
Let (X(t): t > 0) be a stationary time-homogeneous, pure-jump Markov process with state space 
{1,2,3,4} and Q matrix 


—1 1 0 0 
O -l-a 1 Q 

Q 1 0 -1 0 : 
0 0 1 -—1 


where a > 0. Find the Q matrix of the time-reversed process. Sketch the rate transition diagram 
of both X and the time-reversed process. 


5.2. Reversibility of a three state Markov process 
Consider the continuous-time Markov process with the transition rate diagram shown. Identify all 
possible values of (a, b,c) € R? which make the process reversible. 


5.3. A link with heterogeneous call requirements 

Let M = K x L, where K and L are strictly positive integers, and let Às, Ap > 0. A bank of M 
channels (i.e. servers) is shared between slow and fast connections. A slow connection requires the 
use of one channel and a fast connection requires the use of L channels. Connection requests of each 
type arrive according to independent Poisson processes, with rates As and Ap respectively. If there 
are enough channels available to handle a connection request, the connection is accepted, service 
begins immediately, and the service time is exponentially distributed with mean one. Otherwise, 
the request is blocked and cleared from the system. 

(a) Sketch a transition rate diagram for a continuous-time Markov model of the system. 

(b) Explain why there is a simple expression for the equilibrium probability distribution, and find 
it. (You don’t need to find an explicit expression for the normalizing constant.) 

(c) Express Bs, the blocking probability for slow requests, and Br, the blocking probability for 
fast requests, in terms of the equilibrium distribution. 


5.4. Time reversibility of an M/GI/1 processor sharing queue 

A processor sharing server devotes a fraction 1/n of its effort to each customer in the queue when- 
ever there are n customers in the queue. Thus, if the service requirement of a customer arriving 
at time s is x, then the customer departs at the first time s’ such that ie 1/N(t)dt = x, where 
N (t) denotes the number of customers in the system at time t. It turns out that if the queue is 
stable then N is time-reversible, and its equilibrium distribution depends only on the load p. In 
particular, the departure process is Poisson. This result is verified in this problem for a certain 
class of distributions of the service requirement X of a single customer. (a) Is N Markovian? If 
not, explain what additional information would suffice to summarize the state of the system at a 
time t so that, given the state, the past of the system is independent of the future. 

(b) To arrive at a Markov model, restrict attention to the case that the service requirement X 
of a customer is the sum of k exponentially distributed random variables, with parameters p; for 
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1<i<k. Note that E[X] =1/p1 +:::+1/pj and p=AE[X]. Assume p < 1. Equivalently, X is 
the time it takes a pure birth Markov process to move from state 1 to state k+1 if the birth rate in 
state ¿ is u;. Thus, each customer moves through k stages of service. Given there are n customers 
in the system, a customer in stage i of service completes that stage of service at rate p;/n. Let 
N;(t) denote the number of customers in the system that are in stage i of service at time t. Let 
O(t) = (Ni(t),..., Ny(t)). Then O is a Markov process. Describe its transition rate matrix Q. 

(c) The process © is not reversible. Show that the equilrium distribution of O is given by 
T(ni,... nk) = (1 -PP (m 1 ---a,*, where n = ni +---+n, and a; = (1/45)/(1/11+:--+ 
1/px). At the same time, show that the time reverse of O has a structure similar to O, except that 
instead of going through the stages of service in the order 1,2,...,k, the customers go through 
the stages of service in the order k,k —1,...,1, with the service requirement for stage i remaining 
exponentially distributed with paramenter pu. (It follows that then number of customers in the 
system in equilibrium has distribution P|N(t) = n] = (1—p)p”.) (d) Explain why N(t) is reversible 
for the class of service times used. (Note: If u; = k for 1 < i < k, then as k — o, the service 
time requirement approahes the determinitic service time X = 1. By using a more general Markov 
process to model X, any service time distribution can be approximated, and still N is reversible.) 


5.5. A sequence of symmetric star shaped loss networks 

Consider a circuit switching system with M > 2 links, all sharing an endpoint at a central switching 
node. Suppose the set of routes used by calls is the set of all (unordered) pairs of distinct links. 
There are thus es ) routes. Suppose that calls arrive for any given route according to a Poisson 
process of rate y = 4/(M — 1). Suppose that the call holding times are independent and exponen- 
tially distributed with mean one. Finally, suppose that each link has capacity C = 5, so that a call 
offered to a route is blocked and discarded if either of the two links in the route already has 5 calls 
in progress. (a) What Markov chain describes the system? Describe the state space and transition 
rates. Is the chain reversible? (b) What is the conservative upper bound on the blocking probabil- 
ity for a given link, and what is Whitt's lower bound on the call acceptance probability. Consider 
the cases M = 2,3,4 and M — oo. Is the bound exact for any of these values? (c) Derive the 
reduced load approximation for the blocking probability of a link. Making an independence assump- 
tion, derive an approximation for the probability that a call is accepted. Again, consider the cases 
M =2,3,4and M — oo. Comment on the accuracy of the approximation for different values of M. 


Note: Solutions are not provided for the remainder of the problems in this section. 


5.6. A simple open Jackson network 

Consider the Jackson type network (network of single server queues, independent exponentially 
distributed service times with queue-dependent means, exogenous arrival processes are Poisson and 
Markov random routing is used) shown in Figure 5.3. The service rates at the three stations are 1, 
2, and 1 respectively, and the exogenous arrival rate is y. At the three points indicated, a customer 
randomly chooses a path to continue on, each of two choices having probability 0.5. (a) Determine 
the range of y for which the system is positive recurrent. (b) Determine the mean time in the 
network for a typical customer. (c) Each customer exits at either point A or point B. Find the 
probability that a typical customer exits at point A. (Does your answer depend on v?) (d) Indicate 
how the network can be embedded into an equivalent closed queueing network. 


5.7. A tale of two cities 
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Figure 5.3: A three queue Jackson network 


Two cities are connected by two one-way channels, each of capacity C bits per second. Each 
channel carries packets which are initiated according to a Poisson process at rate a packets per 
second. Assume that packet lengths are independent, exponentially distributed with mean length 
ut bits. It is proposed that the two channels be replaced by a single half-duplex channel (capable 
of carrying messages in either direction, but only one direction at a time) of capacity C’. How large 
must C” be in order to achieve smaller mean waiting times than for the original network? (Model 
links by single server queues.) 

5.8. A tale of three cities 

Two communication networks are being considered to link three cities. The first has six one-way 
channels of capacity C bps, one for each ordered pair of cities. The second has three one-way 
channels of capacity 2C bps each, and the channels connect the three cities in a ring. In the first 
network packets need only one hop while in the second network some of the packets need travel 
two hops. Assume that for any ordered pair of cities, packets originate at the first city destined 
for the second city according to a Poisson process of rate a. Thus there are six types of packets. 
Assume that packets are exponentially distributed in length with mean length y”! bits. 

(a) Give a full Markov description for each of these two networks. 

(b) Compute the mean time in the network. (If you need to make an independence approximation 
at any point, go ahead but state that you’ve done so.) Under what conditions on a and C will the 
mean time in the first network be smaller than the mean time in the second network? 

5.9. A cyclic network of finite buffer queues 

K jobs circulate in a computer pipeline with N stages, as shown in Figure 5.4. Each stage is assumed 


1 2 3 4 N 
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Figure 5.4: A recirculating pipeline 


to have one server and storage space for B jobs. Service at any nonempty stage is completed at rate 
one whenever the next stage is not full, and at rate zero otherwise (this is called ”communication 
blocking”). Define n(N, K, B) to be the mean rate at which jobs pass through a given stage in 
equilibrium. (a) Find n(W, K, B) for B > K. (b) Find n(N, K,1). (Hint: show that all states 
are equally likely in equilibrium. The rotational symmetry of the system is not enough.) (c) Let 
n*(B) = limy>o maxín(N, K, B) : K > 1}. Find ņn*(+c0) and n*(1). Note: Recent work on 
interacting particle systems has allowed a determination of 7*(B) for all B. 
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Chapter 6 


Calculus of Deterministic Constraints 


If a communication link is supposed to guarantee a maximum delay to a traffic flow, then the 
traffic flow must also satisfy some constraint. In this section a natural calculus of deterministic 
constraints is examined. In the first subsection the basic (0, p) constraints, and the resulting 
performance bounds for a single server queue are discussed. In the second section, a more general 
notion of deterministic constraint is explored. It is shown that token bucket filters are the minimal 
delay regulators that produce (ø, p)-upper constrained traffic streams, and more general regulators 
are also considered. The third section focuses on applications of service curves, which offer a flexible 
way to specify a lower bound on the output of a server as a function of the input. Given a constraint 
on the input traffic and a service curve, a maximum delay can be computed for any server that 
conforms to the service curve. Two applications are discussed: bounding the maximum delay in a 
priority queue, and scheduling service at a constant rate link with multiple input streams in order 
to achieve a specified service curve for each input. 

This chapter is based largely on the presentation in C.-S. Chang [5]. The basic calculus of (ø, p) 
constraints was first given in the seminal work of R. L. Cruz [8, 9]. 


6.1 The (o, p) Constraints and Performance Bounds for a Queue 


For simplicity, the case of equal length packets transmitted in discrete time will be considered. 
A cumulative arrival process A is a nondecreasing, integer-valued function on the nonnegative 
integers Zy such that A(0) = 0. For each integer t > 1, A(t) denotes the number of arrivals in slots 
1,2,...,t. The number of arrivals at time t, denoted by a(t), is given by a(t) = A(t) — A(t — 1). 
A cumulative arrival process A is said to be (ø, p)-upper constrained if A(t) — A(s) < o + p(t — s) 
for any integers 0 < s < t. This is denoted as A ~ (ø, p). In these notes p and ø are taken to be 
integer valued. 

A popular way to regulate data streams is through the use of a token bucket filter. A token 
bucket filter (with no dropping) with parameters (ø, p) operates as follows. The filter has a packet 
queue with unlimited storage capacity, and it has a token bucket (implemented as a counter). 
The events at an integer time occur in the following order. First new packet arrivals are added 
into the queue and p new tokens are added to the token bucket. Then as many of the packets 
immediately depart as possible, with the only restriction being that each departing packet take one 
token. Finally, if there are more than o tokens in the queue, some tokens are dropped to bring the 
total number of tokens to ø. 
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A token bucket filter with parameters (0, p) is a (ø, p) regulator, meaning that for any input 
process A the output process B is (ø, p)-upper constrained. Indeed, since at most ø tokens are in 
the bucket just before time s + 1, and since p(t — s) additional tokens arrive in slots s + 1,...,t, 
at most o + p(t — s) packets can depart from the filter in those slots. So the departure process 
from the token bucket filter is indeed (ø, p)-upper constrained. It is shown in the next section that 
among all FIFO servers producing (ø, p)-upper constrained outputs, the token bucket filter requires 
the least delay for each packet. 

If constrained flows are merged, the output process is also constrained: 


Ai ~ (05, pi) => SA, N DD 05, X pi) 


The remainder of this section deals with a queue with a constant rate server, and examines 
performance bounds implied if the input process is (ø, p)-upper constrained. First the queue process 
is defined. Let C be a positive integer and let A denote a cumulative arrival process. Suppose the 
arrival process is fed into a queue with a server that can serve up to C packets per slot. Think of 
the events at an integer time as occurring in the following order. First arrivals, if any, are added to 
the queue, then up to C customers in the queue are instantaneously served, and then the number 
of customers in the queue is recorded. Since we take service times to be zero, the amount of time a 
customer spends in the queue is the same as the amount of time the customer spends in the system, 
and is simply called the delay of the customer. The queue length process is recursively determined 
by the Lindley equation: 


alt +1) = (a(t) + a(t+1) -= C)+ 


with the initial condition q(0) = 0. As can be shown by induction on t (and as seen earlier in the 
course) the solution to the Lindley equation can be written as 


oe le) Ce 8) (6.1) 


for all t > 0. The cumulative output process B satisfies, by (6.1), 


Bit) A(t) — alt) 


min A(s) + C(t—s) all t>0. 
0<s<t 


Suppose now that the input A is (ø, p)-upper constrained. Then, if p > C, (6.1) immediately 
implies that q(t) < o for all t. Conversely if C = p and q(t) < o for all t, then A ~ (0, p). 

Next, upper bounds on the length of a busy period and on packet delay are derived, assuming 
that the input A is (ø, p)-upper constrained. Given times s and t with s < t, a busy period is said 
to begin at s and end at t if q(s — 1) = 0, a(s) > 0, g(r) > 0 for s < r < t, and q(t) = 0. We define 
the duration B of the busy period to be B = t — s time units. Given such a busy period, there 
must be C departures at each of the B times [s,...,t— 1}, and at least one packet in the queue at 
time t— 1. Thus at least CB +1 packets must arrive at times in {s,...,t—1} in order to ”sustain” 
the busy period. Since A is (ø, p)-upper constrained, at most o + pB packets can arrive in those 
slots. Thus it must be that CB +1 < o + pB which yields B < 2". But since B is an integer it 


mre 
must be that B < 133). 
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The delay of a packet in the queue is the time the packet departs minus the time it arrives. 
The delay of any packet is less than or equal to the length of the busy period in which the packet 
arrived. Thus, 
o=1 
E 

=p 
(for any service order). (Note: If one unit of service time is added, the total system time would be 
bounded by 


d< | 


o=1 oO 
d+1< lg—,/+1s ap! 
which is the bound given in [5], although [5] uses a different definition of the length of a busy 
period. 

A tighter bound can be given in case the order of service is first in first out (FIFO). If a packet 
has a nonzero waiting time, then it is carried over from the time it first arrived to the next time 
slot. The total number of packets carried over, including that packet, is less than or equal to o, 
as shown above. Thus, assuming FIFO order, since at least C' of these q packets depart in each of 
the subsequent slots until all q are served, the delay for the packet is less than or equal to dryro, 
given by 

drrro = [0 /C!. 
(Note: This delay also does not include an extra unit of time for service.) 

In order to analyze networks, it is important to find constraints on the output of a device, 
because the output of one device could be the input of another device. A constraint on the output 
of a device is readily found if the input satisfies a known constraint and if a bound on the maximum 
delay of the device is known. Suppose that the input A to a device is (ø, p)-upper constrained, 
and that the maximum delay in the device is d. Then for s < t, any packets that departs from the 
device at a time in (s+1,...,t) must have arrived at one of the t— s +d times in [s+1—d,...,t). 
Therefore, B(t) — B(s) < A(t) — A(s — d) < o + p(t — (s — d)) = o + pd + p(t — s). Therefore, B is 
(o + pd, p)-upper constrained. 

Consider, for example, a queue with a server of rate C > 1 and input arrival process A. The 
cumulative output process B does not depend on the order of service, so no matter what the order 
of service, the output process is (o + pdrrro, p)-upper constrained. 


6.2  f-upper constrained processes 


The notion of (a, p) constraints has a natural generalization. The generalization is useful not only 
for the sake of generalization, but it helps in the analysis of even basic (ø, p) constrained traffic. Let 
f be a nondecreasing function from Z4 to Z4}. A cumulative arrival process A is said to be f-upper 
constrained if A(t) — A(s) < f(t—s) for all s,t with 0 < s < t. For example, if f(t) = 0 + pt, then 
A is f-upper constrained if and only if it is (ø, p)-upper constrained. 

Rearranging the definition yields that A is f-upper constrained if and only if A(t) < A(s) + 
f(t— s) for 0 < s < t. Equivalently, A is f-upper constrained if and only if A < Ax f, where for 
two functions f and g defined on Z,, f xg is the function on Z, defined by 


(590) mn als) + I~): (6.2) 


Note that f xg is a convolution in the (min, plus) algebra. That is, (6.2) uses a minimum rather 
than addition (as in an integral), and (6.2) uses addition rather than multiplication. 
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It turns out that some functions f can be reduced, without changing the condition that an 
arrival process is f-upper constrained. For example, changing f(0) to zero has no effect on the 
requirement that A is f-upper constrained, because trivially A(t) — A(t) < 0 anyway. Moreover, 
suppose A is f-upper constrained and s,u > 0. Then A(s + u) — A(s) < f(u), but a tighter bound 
may be implied. Indeed, if n > 1 and u is represented as u = u1 +--+ Un where u; > 1, integer, 
then 


A(s+u)—A(s) = (A(s +u) — A(s)) + (A(s + u +u) — A(s +u) 
+ + (A(s +u +--+ un) — A(s + ui t:e + un)) 
fur) + F(u2) ++ flun) 


So A(t) — A(s) < f*(t— s) where f*, called the subadditive closure of f, is defined by 


IA 


0 ifu=0 
{ min{ f(w1)+---+ f(tun):n>1,u;j > 1 for each i, and ui+---+tn=u} ifu>1 


f(u) = 
The function f* has the following properties. 

a) <f 

b) A is f-upper constrained if and only if A is f*-upper constrained 

c) f* is subadditive; f*(s +t) < f*(s) + f*(t) all s,t > 0. 

d) If g is any other function with g(0) = 0 satisfying (a) and (c), then g < f*. 


Recall that the output of a token bucket filter with parameters (ø, p) is (a, p)-upper constrained 
for any input. More generally, given a nondecreasing function f from Z+ to Z4, a regulator for f 
is a device such that for any input A the corresponding output B is f-upper constrained. We shall 
only consider regulators that output packets only after they arrive, requiring B < A. 

Roughly speaking, regulators (that don’t drop packets) delay some packets in order to smooth 
out the arrival process. There are many possible regulators for a given arrival sequence, but it is 
desirable for the delay in a regulator to be small. A regulator is said to be a maximal regulator for 
f if the following is true: For any input A, if B is the output of the regulator for input A and if 
B is a cumulative arrival process such that B < A (flow condition) and B is f-upper constrained, 
then B < B. Clearly if a maximal regulator for f exists, it is unique. 


Proposition 6.2.1 A mazimal regulator for f is determined by the relation B = Ax f*. 
Proof. Let A, B, and B be as in the definition of maximal regulator. Then 
B=B+f"*<Axf'=B (6.3) 
The first equality in (6.3) holds because Bis f-upper constrained. The inequality holds because 


xf* is a monotone operation. The final equality holds by the definition of B. a 


Corollary 6.2.2 Suppose fı and fz are nondecreasing functions on Zy with fı(0) = fa(0) = 0. 
Two maximal regulators in series, consisting of a maximal regulator for fı followed by a maximal 
regulator for fo, is a maximal regulator for fx fo. In particular, the output is the same if the order 
of the two regulators is reversed. 


116 


Proof. Let A be the input to the first regulator and let B be the output of the second regulator. 
Then 


B = (Ax ff) * fo = Ax (fx fa) = Ax (fx fo)”, (6.4) 


where the last equality in (6.4) depends on the assumption that fı(0) = f2(0) = 0. The first part 
of the corollary is proved. The second part follows from the uniqueness of maximal regulators and 
the fact that fi * fe = fo * fi. E 


Proposition 6.2.3 The token bucket filter with parameters (o, p) is the maximal (o, p) regulator. 


Proof. A key step in the proof is to represent the state of a token bucket filter as a function 
of the queue length process for a queue with constant server rate p packets per unit time. Let A 
denote a cumulative arrival process, and let q denote the corresponding queue length process when 
A is fed into a queue with a server of constant capacity p packets per time unit. Therefore, 

q(t) = max[A(t) — Als) — plt— s)]. 

Then the size of the packet queue for a (ø, p) token bucket filter is (q(t) — 7)4, and the number 
of tokens in the the filter is (a — q(t))+. See Figure This representation can be readily proved by 
induction on time. 

Therefore, if B is the cumulative departure process from a (ø, p) token bucket filter with input 
A, then B is given by 


Bit) = A(t) — (q(t) — 0)+ 
= A(t) — (gua A(t) — A(s) — [0 + p(t — a). 


0<s<t 


min faw, min, A(s) + o + p(t — 5} 


That is, B= f* x A for f(t) =o + pt, which concludes the proof. E 


6.3 Service Curves 


So far we have considered passing constrained processes into a maximal regulator. Examples of 
maximal regulators include a token bucket filter, or a queue with fixed service rate. The output 
for these servers is completely determined by the input. In practice, packet streams can be fed 
into many other types of servers, which often involve interaction among multiple packet streams. 
Examples include a station in a token ring network or a link in a communication network with 
a service discipline such as pure priority or weighted round robin. The exact service offered to a 
particular stream thus cannot be fixed in advance. It is thus useful to have a flexible way to specify 
a guarantee that a particular server offers to a particular stream. The definition we use naturally 
complements the notion of f-upper constraints discussed above. 

A service curve is a nondecreasing function from Z+ to Z+}. Given a service curve f, a server is 
an f-server if for any input A, the output B satisfies B > Ax f. That is, B(t) > mino<s<:{A(s) + 
f(t—s)}. 
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Some comments about the definition are in order. First, note that the notion of f-server involves 
an inequality, rather than a complete specification of the output given the input as in the case for 
regulators. Second, in these notes we only consider servers such that the flow constraint B(t) < A(t) 
is satisfied, and it is assumed that A(0) = 0 for all cumulative arrival streams. Taking s = t in 
the above equation implies that B(t) > A(t) + f(0). This can only happen if f(0) = 0, so that 
when we discuss f-servers we will only consider functions f with f(0) = 0. Third, the definition 
requires that the output inequality is true for all input processes (it is “universal” in A). In some 
applications (not discussed in these notes) a server might be an f server only for a specified class 
of input processes. 

Examples For an example of a service curve, given an integer d > 0, let Og denote the function 


O fort<d 
+oo fort>d 


Oa(t) = { 


Then a FIFO device is an Og-server if and only if the delay of every packet is less than or equal 
to d, no matter what the input process. A queue with constant service rate C is an f server for 
f(t) = Ct. A leaky bucket regulator is an f server for f(t) = (o + pt)Ig¢>1;. More generally, the 
maximal regulator for a function f is an f*-server. 

The following proposition describes some basic performance guarantees for a constrained input 
process passing through an f-server. Suppose an f¡-upper constrained process A passes through 
an fo-server. Let dy = max+>0(f; (t) — fo(t)) and dy = min{d > 0: f(t) < fo(t+d) for all t > 0). 
That is, dy is the maximum vertical distance that the graph of ff is above f2, and dy is the 
maximum horizontal distance that the graph of fə is to the right of ff. 


Proposition 6.3.1 The queue size A(t) — B(t) is less than or equal to dy for any t > 0, and if 
the order of service is FIFO, the delay of any packet is less than or equal to dy. 


Proof. Let t > 0. Since the server is an fo-server, there exists an s* with 0 < s* < t such 
that B(t) > A(t — s*) + fo(s*). Since A is f¡-upper constrained, A(t) < A(t — s*) + ff(s*). Thus, 
A(t) — B() < f(s") — fals*) < dy. i 

To prove the delay bound, suppose a packet arrives at time t and departs at time t > t. Then 
A(t) > B(t— 1). Since the server is an f-server, there exists an s* with 0 < s* < t — 1 such that 
B(t-1) > A(s*) + fo(t —1— s*). Because A(t) > B(t— 1), it must be that 0 < s* < t. Since Aj 
is f¡-upper constrained, we thus have 


Ae +t fit=s) > A(t) > BE=1) > Ale) + fat- 1- s*), 
so that fı(t— s*) > fa(t— 1 -— s*). Hence, t— 1 — t < dy, so that t— t < dy. l 


Consider a queue with a server of constant rate C which serves input streams 1 and 2, giving 
priority to packets from input 1, and serving the packets within a single input stream in FIFO 
order. Let A; denote the cumulative arrival stream of input 7. 


Proposition 6.3.2 (See [5, Theorem 2.3.1]). If Ai is fi-upper constrained, the link is an fo-server 
for the type 2 stream, where fa(t) = (Ct — f,(t))*. 


Example For example, suppose A; is (0;, p;) constrained for each i. Then fa(t) = ((C — pı)t — 
01)+. Applying Proposition 6.3.1 to the input 2 yields that the delay for any packet in input 2 is 
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less than or equal to aa Since packets in input 1 are not affected by the packets from input 2, 
the delay for any packet in input 1 is less than or equal to [@]. 

In comparison, if the queue were served in FIFO order, then the maximum delay for packets 
from either input is [242]. Conclude that if 0, is much smaller than da, the delay for input 1 
packets is much smaller for the priority server than for the FIFO server. 

The final topic of the section is the problem of scheduling service for multiple input streams on 
a constant rate link to meet specified service curves for each input. The scheduling algorithm used 
is the Service Curve Earliest Deadline (SCED) algorithm. Suppose the ith input has cumulative 
arrival process A; which is known to be g;-upper constrained, and suppose that the ith input wants 
to receive service conforming to a specified service curve f;. Suppose the link can serve C packets 
per unit time. Given C, the functions g; and service curves f; for all 7, the basic question addressed 
is this. Is there a scheduling algorithm (an algorithm that determines which packets to serve at 
each time) so that the service for each input A; conforms to service curve f;? 

The SCED algorithm works as follows. For each input stream i let N; = A; x fi. Then N;(t) 
is the minimum number of type i packets that must depart by time t in order that the input i see 
service curve f;. Based on this, a deadline can be computed for each packet of input 7. Specifically, 
the deadline for the kth packet from input i is the minimum t such that N;(t) > k. Simply put, if 
all packets are scheduled by their deadlines, then all service curve constraints are met. 

In general, given any arrival sequence with deadlines, if it is possible for an algorithm to meet 
all the deadlines, then the earliest deadline first (EDF) scheduling algorithm can do it. The SCED 
algorithm, then, is to put deadlines on the packets as described, and then use earliest deadline first 
scheduling. 


Proposition 6.3.3 Given gi, fi for each i and capacity C satisfying 


n 


X gi x fi)(t) < Ct all t. (6.5) 


i=1 


the service to each input stream i provided by the SCED scheduling algorithm conforms to service 
curve fi. 


Proof. Fix a time tọ > 1. Imagine that all packets with deadline tọ or earlier are colored red, 
and all packets with deadline tọ + 1 or later are white. For any time t > 0, let qo(t) denote the 
number of red packets that are carried over from time t to time t+ 1. Since EDF is used, red 
packets have pure priority over white packets, so q. is the queuelength process in case all white 
packets are simply ignored. It must be proved that qo(to) = 0. For 0 < s < to — 1, the number of 
red packets that arrive from stream ¿ in the set of times {s+1,--- ,to} is (Ni(to) — Aj(s))+. Also, 
Ni(to) < Ai(to). Therefore 


do(to) = max (Eos = ato) (lo): 


0<s<to 
For any s with 1 < s < to, 


N; (to) = (Ai x fi) (to) < (Ai * gi * fi) (to) = min Ailu) + (gix fi)(to — u) < Ai(s) + (gi * fi) (to — 8) 


It follows that 
S (Ni (to) — Ai(s))+ < Divi x fi)(to — 8) € C(to — 5), 


1 


so that qo(to) = 0, and the proposition is proved. E 


6.4 Problems 


6.1. Deterministic delay constraints for two servers in series 
Consider the network shown. There are three arrival streams. Suppose that A; is (0;, pi)-upper 
a a e 
A 3 Se | ay 
FIFO A PRIORITY TO A 
3 
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constrained for each ¿ and that pı + pa < Cı and pi + p3 < Co. The first server is a FIFO server, 
and the second server gives priority to stream 3, but is FIFO within each class. For all three parts 
below, express your answers in terms of the given parameters, and also give numerical answers for 
the case (0;, pi) = (4,2) for 1 < i < 3 and C1 = C2 = 5. 

(a) Give the maximum delay dı for customers of the first queue. (Your answer should be finite 
even if pi + pa = C¡ due to FIFO service order). (b) Find a value of ø such that B; is (ø, p1)-upper 
constrained. (Seek the smallest value you can.) 

(c) Find an upper bound on the delay that stream 1 suffers in the second queue. 


6.2. Calculation of an acceptance region based on the SCED algorithm 
Let 

gi(t)=20+¢ filt) = 5(t—4)4 

glt) =8-+4t falt) = 6(t — 20), 


(a) Verify that 
va f 0 ift=0 
si (t) =f g(t) ift>1 


(b) Sketch g* x fi for i = 1,2 on two separate graphs. These functions are piecewise linear. Label 
the breakpoints (places the slopes change), the values of the functions gř x f; at the breakpoints, 
and the values of slope between breakpoints. 

(c) Suppose that nı + na streams share a constant bit rate link of capacity C = 100 such that 
for i = 1,2, there are n; streams that are g;-upper constrained, and that each of these n; streams 
require the link to appear as an fi-server. Using Service Curve Earliest Deadline first (SCED), it 
is possible to accommodate the streams if (n1, n2) satisfies >>, nig x fi < Ct for all t. Find and 
sketch the region of admissible (n1, n2) pairs. (Hint: It is enough to check the inequality at the 
breakpoints and at t — oo. Why?) 


6.3. Serve longer service priority with deterministically constrained arrival processes 
Suppose two queues are served by a constant rate server with service rate C, which serves the 


longer queue. Specifically, suppose a discrete-time model is used, and after new customers arrive 
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in a given slot, there are C potential services available, which are allocated one at a time to the 
longer queue. For example, if after the arrivals in a slot there are 9 customers in the first queue, 
and 7 in the second, and if C = 4, then 3 customers are served from the first queue and 1 from the 
second. Suppose the arrival process A; to queue i is (0;, pi)-upper constrained, for each i, where 
01,02, p1,p2, and C are strictly positive integers such that pj + pa < C. 


Al 


N c 


Az serve 
j longer 


(a) Find a bound on the maximum number of customers in the system, carried over from one slot 
to the next, which is valid for any order of service within each substream. 

(b) Find a bound on the maximum delay which is valid for any order of service within each sub- 
stream. (The bound should be finite if p1 + p2 < C.) 

(c) Suppose in addition that customers within each stream are served in FIFO order. Find an 
upper bound for the delay of customers in stream 1 which is finite, even if pi + pa = C. Explain 
your reasoning. 
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Chapter 7 


Graph Algorithms 


In class we covered: 
(i) basic graph terminology 


(ii) the Prim-Dijkstra and Kruskul algorithms for the minimum weight spanning tree problem in 
a directed graph 


(iii) the Bellman-Ford and Dijkstra algorithms for the shortest path problem 


Those topics are not covered in these notes: the reader is refered to Bertsekas and Gallager, Data 
Networks, for those topics. 


7.1 Maximum Flow Problem 


A single-commodity flow network is a collection N = (s,t,V,E,C), where (V, E) is a directed 
graph, s and t are distinct nodes in V, and C = (C(e) : e € E) is a vector of nonnegative link 
capacities. We write e = (i, j) if e is the link from node 7 to node j, in which case C(e) can also be 
written as C(i,7). Let f = (f(e) : e € E) such that 0 < f(e) < C(e) for all links e. The net flow 
into a node j is given by 

IS Sa Ore (7.1) 


i(i, j)EE k:(j,k)€ E 


and the net flow out of node j is minus one times the flow into node j. The net flow into a set of 
nodes is the sum of the net flows into the nodes within the set. If the net flow into each node other 
than s and t is zero, then f is called an s — t flow. The value of an s — t flow is defined to be the 
net flow out of node s, or, equivalently, the net flow into node t. 

An s—t cut for N is a partition (A: B) of the set of nodes (meaning that AUB = V and ANB = 
Ø) such that s € A and t € B. The capacity of such a cut is C(A, B) => u jjeraca jeg C(t, j) The 
famous min-flow, max-cut theorem, is stated as follows. 


Theorem 7.1.1 (Max-Flow, Min-Cut Theorem) The maximum value of s — t flows is equal 
to the minimum capacity of s — t cuts. 
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We shall give a proof of the theorem in the remainder of this section. On the one hand, for any 
s — t flow f and any s — t cut (A: B) 


value(f) = SS aD- Y ai 
(i, j)EE:iE€A,jEB (j i)EE: EA, jEB 


y C(i, j) = C(A, B) (7.2) 


(i j)EE:iE€A,jEB 


IA 


The inequality in (7.2) is a consequence of the fact that f(i,7) < C(i,j) and f(j,7) > 0 for allie A 
and j € B. Thus, the value of any s — t flow is less than or equal to the capacity of any s — t cut. 
It remains to be shown that there exists an s — t flow and an s — t cut such that the value of the 
flow is equal to the capacity of the cut. 

Let f be an s — t flow with maximum value. Such a flow exits because the set of s — t flows 
is compact and value(f) is a continuous function of f. Without loss of generality, suppose that 
E = V x V (simply set C(e) = 0 for links not originally in the network.) Call a link (i, j) good if 
fij) < C(i,j) or f(j,i) > 0. Intuitively, a link is good if the net link flow f(i,j) — f(3,1) can 
be increased without violating the capacity constraints. Let A denote the set of nodes i in V such 
that there is a path from s to ¿ consisting of good links, and let B = V — A. Note that t ¢ A, 
for otherwise the flow vector f could be modified to yield a flow with strictly larger value. In 
addition, by the definition of good link and of A, f(i, j) = C(i, j) and f(j,i) = 0 for all i € A and 
all j € B. Thus, the inequality in (7.2) holds with equality for this specific choice of cut (A, B). 
This completes the proof of the min-flow, max-cut theorem. 

The above proof suggests an algorithm for computing an s — t flow with maximum value. 
Starting with an arbitrary flow f, one seeks a path from s to t using only good links. If no such 
path can be found, the flow has maximum value. If a path is found, a new flow with a strictly 
larger value can be found. An algorithm can in fact be found (called the Malhotra, Kumar and 
Maheshwari algorithm) to find a maximum flow in O(|V|?) steps. A variation of the algorithm can 
also find a flow with minimum weight, among all flows with a specified value, when nonnegative 
weights are assigned to the links of the graph, in addition to the capacities. 


7.2 Problems 


7.1. A shortest path problem 

A weighted graph with a single source node s is shown in Fig. 7.1. The weight of each edge is 
assumed to be the same in either direction. (a) Indicate the tree of shortest paths from s to all 
other nodes found by Dijkstra’s shortest path algorithm. (b) Is the tree of shortest paths from s for 
the graph unique? Justify your answer. (c) How many iterations of the synchronous Bellman-Ford 
algorithm are required to find shortest length paths from the source node s to all other nodes of 
this graph? 

7.2. A minumum weight spanning tree problem 

Consider again the graph of Figure 7.1. Find the minimum weight (undirected) spanning tree found 
by Dijkstra’s MWST algorithm. Is the MWST for this graph unique? Justify your answer. 

7.3. A maximum flow problem 

A flow graph with a source node s and a terminal node t is indicated in Figure 7.2. A line between a 
pair of nodes indicates a pair of directed links, one in each direction. The capacity of each directed 
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Figure 7.2: A weighted graph with terminals s and t. 


link in a pair is the same, and is indicated. Find a maximum value s — t flow and prove that it is 
indeed a maximum flow. 
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Chapter 8 


Flow Models in Routing and 
Congestion Control 


This chapter gives an introduction to the study of routing and congestion control in networks, using 
flow models and convex optimization. The role of routing is to decide which routes through the 
network should be used to get the data from sources to destinations. The role of congestion control 
is to determine data rates for the users so that fairness among users is achieved, and a reasonable 
operating point is found on the tradeoff between throughput and delay or loss. The role of flow 
models is to give a simplified description of network control protocols, which focus on basic issues of 
stability, rates of convergence, and fairness. Elements of routing and congestion control not covered 
in this chapter, include routing tables, messages exchanged for maintaining routing tables, packet 
headers, and so on. 


8.1 Convex functions and optimization 


Some basic definitions and facts concerning convex optimization are briefly described in this section, 
for use in the rest of the chapter. 

Suppose that Q is a subset of R” for some n > 1. Also, suppose that Q is a convez set, which 
by definition means that for any pair of points in (2, the line segment connecting them is also in 
Q. That is, whenever 1,1 € Q, ar + (1 — ajax" € Q for 0 <a < 1. A function f on 2 is a 
convex function if along any line segment in 2, f is less than or equal to the value of the linear 
function agreeing with f at the endpoints. That is, by definition, f is a convex function on (2 if 
f(ax+(1—a)2’) <afí(x)+(1—a) f(x”) whenever 1,1" E€ Qand0<a<1. A concave function f is 
a function f such that — f is convex, and results for minimizing convex functions can be translated 
to results for maximizing concave functions. 

The gradient, Vf, of a function f is a vector valued function, defined to be the vector of first 
partial derivatives of f: 

of 


Oxy 


We assume that V f exists and is continuous. If x € Q and if v is a vector such that x + ev € Q for 
small enough e > 0, then the directional derivative of f at x in the direction v is given by 
tn (2 + 60) — F(2) 
e=>0 € 


= (Vf (0) v= P ¿(ajos (8.1) 


We write x* € argminsen f(x) if x* minimizes f over Q, i.e. if x* € Q and f(a*) < f(x) for all 
TERN. 


Proposition 8.1.1 Suppose f is a convex function on a convex set Q with a continuous gradient 
function Vf. Then x* € argmingen f(x) if and only if (Vf(a*))- (£ — x*) > 0 for alz EQ. 


Proof. (Necessity part.) Suppose x* € argminzeg f(x) and let x € Q. Then 2* + e(x — 
x*) € Q for 0 < e < 1, so by the assumption on 2%, f(x*) < f(x* + e(x — x*)), or equivalently, 
dee a A > 0. Letting e > 0 and using (8.1) with v = x — x* yields (V f(x*))-(2—x*) > 0, 
as required. 

(Sufficiency part.) Suppose x* € Q and that (V f(x*)) - (x — x*) > 0 for all x € Q. Let x € Q. 
Then by the convexity of f, f(ex + (1 — €)a*)) < ef (x) + (1 — e) f (x*) for 0 < e < 1. Equivalently, 
f(z) > f(z*) 4 pA est at oe AE Taking the limit as e — 0 and using the hypotheses about x* 
yields that f(x) > f(a*) + (Vf(a*)) - (a —a*) > f(x*). Thus, 2* € arg mingen f(z). E 


8.2 The Routing Problem 


The problem of routing in the presence of congestion is considered in this section. The flow rates 
of the users are taken as a given, and the problem is to determine which routes should provide the 
flow. In a later section, we consider the joint routing and congestion control problem, in which 
both the flow rates of the users and the routing are determined jointly. The following notation is 
used for both problems. 


J is the set links. 


R is the set of routes. Each route r is associated with a subset of the set of links. We write j € r 
to denote that link 7 is in route r. If all routes use different sets of links, we could simply 
consider routes to be subsets of links. But more generally we allow two different routes to 
use the same set of links. 


A is the link-route incidence matrix, defined by Aj, = 1 if j € r and Ajy = 0 otherwise. 


S is the set of users. Each user s is associated with a subset of the set of routes, which is the set 
of routes that serve user s. We write r € s to denote that route r serves user s. We require 
that the sets of routes for different users br disjoint, so that each route serves only one user. 


H is the user-route incidence matrix, defined by H,, = 1 if r € s and Hs r = 0 otherwise. 
yr is the flow on route r. 


fj is the flow on a link j: fj = a Yr, or in vector form, f = Ay. 
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£s is the total flow available to user s: £s = e sYr, or in vector form, x = Hy. The vector 
x = (zs : s € S) is fixed for the routing problem, and variable for the joint congestion control 
and routing problem considered in Section 8.4. 


Di(f;) is the cost of carrying flow j on link j. The function D; is assumed to be a convex, 
continuously differentiable, and increasing function on R+. The cost associated with flow 
rate fj on link j is Di(f;). 


The routing problem, which we now consider, is to specify the route rates for given values of 
the user rates. It can be written as: 


ROUTE(x,H,A,D): 
min TDS v 
jEJ r:jer 
subject to 
x = Hy 


over 
y 20. 


We say that the vector y is a feasible flow meeting demand zx if y > 0 and x = Hy. Problem 
ROUTE(z,H,A,D) thus concerns finding a feasible flow y, meeting the demand x, which minimizes 


the total cost F(y), defined by 
F(y) = Dj) yr). 
J TJET 

As far as the mathematics is concerned, the links could be resources other than connections 
between nodes, and the set of links in a route could be an arbitrary subset of J. But to be concrete, 
we can think of there being an underlying graph with nodes representing switches in a network, 
links indexed by ordered pairs of nodes, and routes forming paths through the network. We could 
assume each user has a specific origin node and destination node, and the routes that serve a user 
each connect the source node to the destination node of the user. 

A possible choice for the function D; would be 


fj 
D;(fj) = + fjdj (8.2) 
INSI Cj- fj Id 
where Cy is the capacity of link j and d; is the propagation delay on link j. If fj and C} are in units 
of packets per second, if arrivals are modeled as a Poisson process, and if service times are modeled 
as exponentially distributed (based on random packet lengths with an exponential distribution), 
then a link can be viewed as an M/M/1 queue followed by a fixed delay element, and ao 7; is the 


mean system time for the queue. Thus, the mean transit time of a packet over the link j, including 
the propagation delay, is T + dj. Then by Little's law, D;(f;) given by (8.2), represents the 
mean number of packets in transit on link j. Therefore, the total cost, F(y), represents the mean 
number of packets in transit in the network. By Little’s law, the average delay of packets entering 
the network is thus F(y)/ >} >,as5. For fixed user flow rates (x, : s € S), minimizing the cost is 
therefore equivalent to minimizing the delay in the network, averaged over all packets. 
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Figure 8.1: Example of a network showing (a) the network, (b) route flows, and (c) link flows. 


Example Consider the network graph shown in Figure 8.1(a). For this example there is an 
underlying set of 5 nodes, numbered 1 through 5. For this example we take 


J = (1,2), (1,4), 


(2,1), (2,3), (2,5), (3,5), (4,5), (5, 4)), with each link labeled by its endpoints. 
S = {(1,5), (2,4), (2,5)}, with each user labeled by its origin and destination nodes. 
1 


We assume user (1,5) is served by routes (1,2,5), (1,4,5), (1,2,3,5) 
user (2,4) is served by routes (2,1,4), (2,5,4) 
user (2,5) is served by routes (2,5), (2,3,5) 
where each route is labled by the sequence of nodes that it traverses. 


For brevity we write the flow Y, for a route r = (1,2,5) as yi25, rather than as Y(1 25) We 
also write the rate x, for a user s = (1,5) as 71,5, rather than as x(1 5), and the flow fj for a link 
j = (1,2) as f¡2, rather than as f(1 2) An example of an assignment of flow rates to routes, shown 
in Figure 8.1(b), is given by 


yi25 = 3.0, y145 =3.0, y1233 = 0, y214 = 1.0, Yyo54 =0, Ya = 2.0, y235 =0 (8.3) 
which implies the flow rates to the users 
215=00,. £24 = 1.0, z25 = 2.0 (8.4) 
and the link rates f12 = 3, fia = 4, f21 = 1, etc., as shown in Figure 7.1(c). The resulting cost is 
F(y) = D12(3) + Dia4(4) + D2,1(1) + Da,3(0) + Do5(5) + D3,5(0) + Da,5(3) + Ds,4(0). 


We consider, in the context of this example, the effect on the cost of changing the flow rates. If 
y125 were increased to y125 + € for some e > 0, then the cost F(y) would increase, with the amount 
of the increase given by 


AF D12(3 + €) — Di2(3) + Das(5 + €) — Do5(5) 


(Di 3(3) + Do5(5)) + of) 


Similarly, if y145 were increased by e, the cost would increase by e( D1 4(4) + D¿5(3))e + ole). If 


? 


instead, y145 were decreased by e, the cost would decrease by e( D4 ¿(4) + Di 5(3))e + o(e). If two 
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small changes were made, the effect on F(y), to first order, would be the sum of the effects. Thus, 
if y125 were increased by e and y145 were simultaneously decressed by e, that is, if e units of flow 
were deviated to route (1,2,5) from route (1, 4,5), then the change in cost would be 


(cost of 145—125 flow deviation) = «(Dj 2(3) + Da 5(5) — Dj 4(4) — D4,5(3)) + o(e) 


That is, to first order, the change in cost due to the flow deviation is e times the first derivative 
length D4 2(3) + Da (5) of route (1, 2,5) minus the first derivative length Dj ¿(4) + D/ 5(3) of route 
(1, 4,5). Thus, for small enough e, the flow deviation will decrease the cost if flow is deviated from a 
longer route to a shorter one, if the lengths of the routes are given by the sums of the first derivative 
link lengths, D;(f;), of the links in the routes. 


We return from the example to the general routing problem. The observation made about flow 
deviation in the example is general. It implies that a necessary condition for y to be optimal is 
that for any s and any r € s, the rate y, is strictly positive only if r has minimum first derivative 
length among the routes serving s. Since the set of feasible rate vectors y meeting the demand x 
is a convex set, and the cost function F is convex, it follows from elementary facts about convex 
optimization, described in Section 8.1, that the first order necessary condition for optimality is also 
sufficient for optimality. Specifically, the following is implied by Proposition 8.1.1. 


Proposition 8.2.1 A feasible flow y = (yr : r € R) meeting demand x minimizes the cost over all 
such flows, if and only if there exists a vector (As: s € S) so that Dosey Di (fj) > As, with equality 
if yr > 0, whenever r E s eS. 


Given a flow vector y, let 6, denote the first derivative length of route r. That is, 6, = 
Di en Dj rje Yr). Note that if the rate requirement for a user s is strictly positive, i.e. £s > 0, 
then the parameter A, in Proposition 8.2.1 must satisfy A, = Minres ôr. The example and Propo- 
sition 8.2.1 suggest the following algorithm, called the flow-deviation algorithm, for finding an 
optimal flow y. The algorithm is iterative, and we describe one iteration of it. 


The flow deviation algorithm One iteration is described. At the beginning of an itera- 
tion, a feasible flow vector y, meeting the specified demand zx, is given. The iteration finds 
the next such vector, y. 


Stepl. Compute the link flow f; and first derivative link lengths, D;(f;), for j € J. 


Step 2. For each user s, find a route rž serving s with the minimum first derivative link 
length. 


Step 3. Let a € [0,1]. For each user s, deviate a fraction a of the flow from each of the 
other routes serving s to rí. That is, for each s, let 


n=] (1 — a)yr ifr € s and r Æ rž 
4 Yr + Mates Yr’) if r = rs 
Adjust a to minimize F(y). The resulting vector y is the result of the iteration. 


The flow deviation algorithm is not a distributed algorithm, but some parts of it can be im- 
plemented in a distributed way. A user could gather the first derivative link lengths of its routes 
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by having the links along each route signal the information in passing packets. A user could then 
determine which of its routes has the shortest first derivative link length. Synchronization of the 
flow deviation step across users, and in particular, the fact that all users must collectively determine 
a single best value of a, is the main obstacle to distributed implementation of the flow deviation al- 
gorithm presented. However, by having each user respond slowly enough, possibly asynchronously, 
convergence can be achieved. 


It may give insight into the problem to ignore the step size problem, find an algorithm, and 
then address the step size selection problem later. This suggests describing an algorithm by an 
ordinary differential equation. For the routing problem at hand, imagine that all users gradually 
shift their flow towards routes with smaller first derivative link length. Let y(t) denote the vector 
of flows at time t, and let 6,(t) denote the corresponding first derivative link length of route r at 
time t, for each r € R. Assume that the users adjust their rates slowly enough that the variables 
9, (t) can be continuously updated. In the following, we suppress the variable t in the notation, but 
keep in mind that y and (ô, : r € R) are time varying. We've seen that if r and r’ are two routes 
for a given user s, if ô» > 6,, and if y,” > 0, then the cost F(y) can be reduced by deviating flow 
from r’ to r. If the speed of deviation is equal to yr (ôr — ôr), then yr is kept nonnegative, and 
the factor (6, — ôr) avoids discontinuities in case two routes have nearly the same length. Suppose 
such changes are made continuously for all pairs of routes. Let 0 = (6,7: r,r” € R) be a matrix of 
constants such that 0,.,. = 0,1 , > 0 whenever r,r’ € s for some s € S, and Opp = 0 if r and r’ serve 
different users. We will use the constant 0, as the relative speed for deviation of flow between 
routes r and r’. The following equation gives a flow deviation algorithm in ordinary differential 
equation form. 


A continuous flow deviation algorithm 


Yr = `> {Or r Yr (Ors = Or) 4 Or 11 Yr (Or = Or) +} forreses (8.5) 


T'Es 


Because the changes in y are due to transfers between routes used by the same user, we expect 
x(t) = Hy(t), to be constant. Indeed, using the symmetry of 0 and interchanging the variables r 
and r’ on a summation, yields that for any user s, 


Ts = Sy = 2 Se Or! Yr! (Ont =z Ôr)+ = y y Orr Yr (Or T rt) + 


rEs TEs r'Es res r'Es 
A AN 
TES 1 "Es TES r!Es 
= 0 
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Now oe = 6,, so by the chain rule of differentiation, and the same interchange of r and r’ used in 


computing Zs, 


dF (y(t) _ ; 
-a = db 


y y {0r 7 Yr Or (Or! me Oy) 4 a Or r’ Yr Or (Or = dr) +} 


reRr'eR 
E a ye {901 rYrt Or (Spt = Or) + = 01 7 Yy! Ôr (Ôr => Or) +} 
TreRr'ER 
ES y y Or! 7 Yr (Ops = One 
TreRr'ER 
< 0 


Thus, the cost F(y(t)) is decreasing in time. Moreover, the rate of decrease is strictly bounded 
away from zero away from a neighborhood of the set of y satisfying the necessary conditions for 
optimality. It follows that all limit points of the trajectory (y(t) : t > 0) are solutions to the routing 
problem, and F(y(t)) converges monotonically to the minimum cost. 

The step size problem is an important one, and is not just a minor detail. If the users do 
not implement some type of control on how fast they make changes, oscillations could occur. For 
example, one link could be lightly loaded one minute, so the next, many users could transfer flow 
to the link, to the point it becomes overloaded. Then they may all transfer flow away from the 
link, and the link may be very lightly loaded again, and so on. Such oscillations have limited the 
use of automated dynamic routing based on congestion information. In applications, an important 
facet of the implementation is that the flow rates at the links often have to be estimated, and the 
accuracy of an estimate over an interval of length t is typically on the order of 1/vt, by the central 
limit theorem. 

The two routing algorithms considered in this section are considered to be primal algorithms, 
because the flow rates to be found are directly updated. In contrast, in a dual algorithm, the 
primal variables are expressed in terms of certain dual variables, or prices, and the algorithm seeks 
to identify the optimal values of the dual variables. Dual algorithms arise naturally for networks 
with hard link constraints of the form fj < Cj, rather than with the use of soft link constraints 
based on cost functions D;(f;) as in this section. See the remark in Section 8.7 


8.3 Utility Functions 


Suppose a consumer would like to purchase or consume some commodity, such that an amount 
x of the commodity has utility or value U(x) to the consumer. The function U is called the 
utility function. Utility functions are typically nondecreasing, continuously differentiable, concave 
functions on R4}. Commonly used examples include: 


(i) U(x) = Blog x (for some constant 8 > 0) 
(i) U(x) = (12) (2122 — 1) (for some constants a, 3 > 0,a Æ 1) 
(iii) U(x) = A [z? — (z — a)4] (for some constants 8, > 0) 
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(iv) U(x) = — 8 exp(—yzx) (for some constant y > 0) 


The function in (ii) converges to the function in (i) as a > 1. 

If x is increased by a small amount e the utility U(x) is increased by e€U'(x) +o(x). Thus U'(x) 
is called the marginal utility per unit of commodity. Concavity of U is equivalent to the natural 
property that the marginal utility is monotone nonincreasing in zx. 

If a rational consumer can buy commodity at a price À per unit, then the consumer would 
choose to purchase an amount x to maximize U(x) — xX. Differentiation with respect to x yields 
that the optimal value xo; satisfies 


U'(Lopt) < A, with equality if Lope > 0. (8.6) 


It is useful to imagine increasing x, starting from zero, until the consumer’s marginal utility is equal 
to the price A. If U is strictly concave, and if (U’)~! is the inverse function of U’, we see that the 
response of the consumer to price A is to purchase Zap = (U’)~1(A) units of commodity. 

The response function for a logarithmic utility function U (x) = Glog x has a particularly simple 
form, and an important interpretation. The response of a rational consumer with such a utility 
function to a price A is to purchase tops given by Lope = B/A. So AXopt = B for any A > 0. That 
is, such a consumer effectively decides to make a payment of 8, no matter what the price per unit 
flow is. 

Exercise: Derive the response functions for the other utility functions given in (i)-(iv) above. 


8.4 Joint Congestion Control and Routing 


In Section 8.2, the flow requirement «x, for each user s is assumed to be given and fixed. However, if 
the values of the x,’s are too large, then the congestion in the network, measured by the sum of the 
link costs, may be unreasonably large. Perhaps no finite cost solution exists. Congestion control 
consists of restricting the input rates to reasonable levels. One approach to doing this fairly is to 
allow the x,’s to be variables; and to assume utility functions U, are given. The joint congestion 
control and routing problem is to select the y,’s. and hence also the x,’s. to maximize the sum of 
the utilities of the flows minus the sum of link costs. We label the problem the system problem, 
because it involves elements of the system ranging from the users’ utility functions to the topology 
and cost functions of the network. It’s given as follows: 


SYSTEM(U,H,A,D) (for joint congestion control and routing with soft link constraints): 


max Y Us(2s) — Y Di) - yr) 


ses jeJ r:jer 
subject to 
x = Hy 
over 
z, y 20. 


This problem involves maximizing a concave objective function over a convex set. Optimality 
conditions, which are both necessary and sufficient for optimality, can be derived for this joint 
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congestion control and routing problem in a fashion similar to that used for the routing problem 
alone.! The starting point is to note that the change in the objective function due in increasing £r 
by e, for some user s and some route r serving s, is 


(Us(2s) — X Di(f;))e + ole) 


jer 
This implies that if y is an optimal flow, then 


U(r.) < e Di(f;), with equality if yr > 0, whenever r € s € $. (8.7) 
jer 


That is, condition (8.7) is necessary for optimality of x. Due to the concavity of the objective 
function, the optimality condition (8.7) is also sufficient for optimality, and indeed Proposition 
8.1.1 implies that (8.7) is both necessary and sufficient for optimality. 

The first derivative length of a link J, D;( fj), can be thought of as the marginal price (which 
we will simply call price, for brevity) for using link j, and the sum of these prices over j € r for 
a route r, is the price for flow along the route. In words, (8.7) says the following for the routes r 
serving a user s: The price of any route with positive flow is equal to the marginal utility for s, 
and the price of any route with zero flow is greater than or equal to the marginal utility for s. 


8.5 Hard Constraints and Prices 


In some contexts the flow f on a link is simply constrained by a capacity: f < C. This can be 
captured by a cost function which is zero if f < C and infinite if f > C. However, such a cost 
function is highly discontinuous. Instead, we could consider for some small e, 6 > 0 the function 


1 


Dif) = x-f-(C- 9) 
The price D'(f) is given by 
Df) = HS (08). (8.8) 


As €,6 — 0, the graph of the price function D’ converges to a horizontal segment up to C and 
a vertical segment at C, as shown in Figure 8.2. That is, for the hard constraint f < C, the 
price is zero if f < C and the price is an arbitrary nonnegative number if f = C. (This is the 
complementary slackness condition for Kuhn-Tucker constraints. ) 

The routing problem formulated in Section 8.2 already involves hard constraints—not inequality 
constraints on the links—but equality constraints on the user rates zs. The variable As can be viewed 
as a marginal price for the rate allocation to user s. The rate £s of user s could be increased by 
e at cost Ase + o(€) by increasing the flow along one of the routes r serving s with minimum first 
derivative link length As. 

The joint routing and congestion control problem with hard link constraints is similar to prob- 
lem SYSTEM(U,H,A,D), but with the cost functions (D; : j € J) replaced by a vector of capacities, 


"In fact, through the introduction of an additional imaginary link, corresponding to undelivered flow, it is possible 
to reduce the joint congestion control and routing problem described here to a pure routing problem. See [2, Section 
6.5.1]. 
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DO | 


Figure 8.2: Steep cost function, approximating a hard constraint. 


C= (C; ay J): 
SYSTEM(U,H,A,C) (for joint congestion control and routing with hard link constraints): 


max ) > U (az) 


ses 


subject to 
x= Hy, Ay<C 


over 
z, y 20. 


The optimality conditions for SYSTEM(U,H,A,C) are, for some vector u = (uj : j € J), 


u; >20, with equality if f; < Cj, for jeJ (8.9) 


U! (£s) < > iy with equality if y, > 0, whenever r€ s€ 5S. (8.10) 

jer 
Equation (8.10) is identical to (8.7), except uj replaces Di (fj). Equation (8.9) shows that pj 
satisfies the limiting behavior of the approximating function D'(f;) in (8.8), as e, — 0. It is 
plausible that (8.9) and (8.10) give optimality conditions for SYSTEM(U,H,A,C), because in a 
formal sense they are the limit of (8.7) as the soft constraints approach hard constraints. The 
sufficiency of these conditions for optimality is proved in an exercise, based on use of a Lagrangian. 


8.6 Decomposition into Network and User Problems 


A difficulty of the joint congestion control and routing problem is that it involves the users’ utility 
functions, the topology of the network, and the link cost functions or capacities. Such large, hetero- 
geneous optimization problems can often be decoupled, using prices (a.k.a. Lagrange multipliers) 
across interfaces. A particularly elegant formulation of this method was given in the work of Kelly 
[24], followed here. See Figure 8.3. The idea is to view the problem faced by a user as how much 
to pay. Once each user s has decided to pay (, for its rate, the network mechanisms can ignore the 
real utility function Us(1s) and pretend the utility function of user s is the surrogate utility function 
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User 1 


Network 


User s AS 


User n Àn 


Figure 8.3: Coupling between the network and user problems in the system decomposition. 


Vslxs) = 3,log(x;). That is because, as noted in Section 8.3, the response of a user with utility 
function Bslog(xs) to a fixed price As is to pay Ps, independently of the price. The network can 
determine (x, y), using the surrogate valuation functions. In addition, for each user s, the network 
can determine a price As, equal to the minimum marginal cost of the routes serving s: 


> jer Di(fj) in the case of soft link constraints 


As = min 6, where 6, = { (8.11) 


jer Hy in the case of hard link constraints. 


The network reports As to each user s. User s can then also learn the allocation rate x, allocated to 
it by the network, because £s = A;@s. The other side of the story is the user optimization problem. 
A user s takes the price As as fixed, and then considers the possibility of submitting a new value of 
the payment, 3,, in response. Specifically, if the price A, resulting from payment (3, is less than the 
marginal utility U/(8s/As), then the user should spend more (increase Bs). Note that the network 
and user optimization problems are connected by the payments declared by the users (the (3,’s) and 
the prices declared by the network (the ,’s). The network need not know the utility functions of 
the users, and the users need not know the topology or congestion cost functions of the network. 


Input data for the network problem includes the vector of payments 8. but not the utility 
functions. The problem is as follows. 


NETWORK(G,H,A,D) (for joint congestion control and routing with soft link constraints): 


max Y fs log(as) — Dl y Yr) 


ses jEJ r:jer 
subject to 
x = Hy 
over 
z,y z0 


The user problem for a given user s does not involve the network topology. It is as follows. 
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USER; (Us, As): 


over 


Let us prove that if the optimality conditions of both the user problem and the network problem 
are satisfied, then the optimality conditions of the system problem are satisfied. We first consider 
the problem with soft link constraints. The optimality conditions for the user problem are: 


Ut(rs) < As, with equality if xs > 0, for s € S. (8.12) 


The optimality conditions for the network problem are given by (8.7) with U! replaced by V/. Since 
V! (xs) = Be (for this to always be true, we interpret Be to be zero if Bs = x, = 0), the optimality 
conditions for the network problem are thus: 


Bs < > Ds(f4), with equality if yr > 0, whenever r € s € S. (8.13) 
Ls £ 
ger 


The fact, true by (8.12), that U/(x5) < As for all s € S, and the rule (8.11) for selecting As, imply 
that U;(£s) < je Di (fj) whenever r € s € S. Thus, it remains to prove the equality condition 
in (8.7) in case yr > 0. If r €s € S and y, > 0, then zs > 0. Therefore, by (8.13), 


(8.14) 


Also, equality holds in both (8.12) and (8.13), so using (8.14) to eliminate As, we find equality holds 
in (8.7). 
The story is similar for the case of hard link constraints, as we see next. The network problem is 


NETWORK(G,H,A,C) (for joint congestion control and routing with hard link constraints): 


maxX. Bs log(2s) 
ses 
subject to 
x= Hy, Ay<C 
over 


z, y 2 0. 


and the same user problem is used. The optimality condition for the user problem is again (8.12). 
The optimality condition for the network problem is given by (8.9) and (8.10) with U? replaced by 
MES 


uj > 0, with equality if fj < Cj, for je J (8.15) 
Bs < Sip with equality if y, > 0, whenever r € s € S. (8.16) 
Ta Ž 

ger 
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We wish to show that the user problem optimality conditions, (8.12), the network problem 
optimality conditions, (8.15) and (8.16), and the choice of (As : s € S), given in (8.11), imply the 
optimality conditions for the system problem, (8.9) and (8.10). Equation (8.15) is the same as 
(8.9). The proof of (8.16) follows by the same reasoning used for soft constraints. 

In summary, whether the link constraints are soft or hard, if the network computes the prices 
using A, = min;¢, Ôr, and if the optimality conditions of the network and user problems are satisfied, 
then so are the optimality conditions of the system problem. 


8.7 Specialization to pure congestion control 


A model for pure congestion control is obtained by assuming the users in the joint congestion con- 
trol and routing problem are served by only one route each. In this situation we can take the set 
of routes R to also be the set of users., and write x, for the rate allocated to user r. If hard link 
constraints are used, the congestion control problem is the following. 


SYSTEM(U,A,D) (for congestion control with soft link constraints): 


max Y Ura) — Y Did > ar) 


reR jeJ r:jer 


over 
x>0. 


The optimality conditions for this problem are 


Up (xr) < Y D(f), with equality if x, > 0, for all r € R. (8.17) 
jer 


If hard link constraints are used, the congestion control problem is the following. 


SYSTEM(U,A,C) (for congestion control with hard link constraints): 


max y Uae) 


reR 
subject to 
AX<C 
over 
x>0 


The optimality conditions for this problem are 


uj > 0, with equality if f; < Cj, for jeJ (8.18) 
U! (£r) < `> lj, with equality if x, > 0, forre R. (8.19) 
jEr 
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Flow 0 Link 1 Link 2 Link 3 


Flow 1 Flow 2 Flow 3 


Figure 8.4: A three link network with four users 


The corresponding network problem is obtained by replacing the utility function U,(x,) by 
Br log(x,) for each r: 


NETWORK(G,A,C) (for congestion control with hard link constraints): 


max y Br log(zr) 
reR 
subject to 
AX<C 
over 
x>0 


The optimality conditions for this problem are special cases of (8.18) and (8.19) and are given 
by (with f = Az): 


uj > 0, with equality if fj < Cj, for j € J (8.20) 
Br = Tr: oy Hj), forre R, (8.21) 
jer 


The user problem for the congestion control system problem is the same as for the joint routing and 
congestion control problems. Basically, the user cannot distinguish between these two problems. 

Remark We remark briefly about dual algorithms, which seek to find an optimal value for the 
vector of dual variables, u. The primal variables, x, are determined by u through (8.21). The dual 
variable uj is adjusted in an effort to make sure that fj < Cj, and to satisfy the other optimality 
condition, (8.20). Intuitively, if for the current value u(t) of the dual variables, the constraint 
fj < CG; is violated, uj should be increased. If the contraint fj is strictly smaller than C}, then pj 
should be decreased (if it is strictly positive) or be held at zero otherwise. 


Example Consider the following instance of the problem NETWORK((, A,C), for congestion 
control with hard link constraints. Suppose J = {1,2,3} and R = {0,1,2,3}, so there are three 
links and fours users, with one route each. Suppose the route of user 0 contains all three links, and 
the route of user ¿ contains only link 7, for 1 < i < 3, as shown in Figure 8.4. Thus, the optimality 
conditions are given by (8.20) and (8.21). In order for x to be an optimal flow, it is necessary for 
this example that f; = C} for all j, because each link has a single-link user. Therefore, 11, 42, and 
u3 can be any values in Ry, and x; = 1 — xo for 1 <i < 3. Condition (8.21) becomes: 


Bo = to(ui + u2 + u3), f1=1p1, B2= 122, B3 = £363.. 
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Eliminating the p’s and using x; = 1 — xo yields: 


by = 00 ( Bi | Be E B3 i 


1 — zo 1 — 29 1 — zo 


or 


_ Bo 
Bo + B1 + B2 + 83° 

For example, if 8, = 1 for all 7, then z = (0.25,0.75, 0.75, 0.75), and u = (1.333, 1.333, 1.333). If 83 
were increased to two, still with G9 = 61 = G2 = 1, then the new allocations and link prices would 
be x = (0.2,0.8,0.8,0.8) and y = (1.25, 1.25,2.5). Thus, the increase in (3 causes an increase in 
the allocation to user 3 from 0.75 to 0.8. A side effect is that the rates of users 1 and 2 also increase 
the same amount, the rate of user 0 is decreased, the price of link 3 increases substantially, while 
the prices of links 1 and 2 decrease somewhat. 


LO 


8.8 Fair allocation 


The problem SYSTEM(U,A,C) is an example of a pure allocation problem with no costs. The 
allocation x is constrained to be in the convex set {x >0: Ax < C}. A generalization is to replace 
the space of x by a general convex set A. For example, we could let A = {x > 0 : J e Ri < C}. 
This leads to the following generalization of SYSTEM(U,A,C): 


ALLOCATE(U,A) 
max Y Ulan) 
reR 
over 
TEA. 


An optimal solution exists if A is closed and bounded and the functions U, are continuous. By 
Proposition 8.1.1, x* is optimal if and only if 


Y O. (at) (Gr —2t) <0 foralzea. (8.22) 
TreR 


In case U,.(1,) = 3, log x, for all r, for some constants 3, > 0, the optimality condition (8.22) 
becomes 


DA) <0 for all x € A. (8.23) 


reR es 
The term in braces in (8.23) is the weighted normalized improvement of x, over x* for user r. For 
example, if oes = 0.5, we say that x, offers factor 0.5 increase over x*. The optimality condition 
(8.23) means that no vector x can offer a positive weighted sum of normalized improvement over 
x*. A vector x* that satisfies (8.23) is said to be a proportionally fair allocation, with weight vector 
B. 

l-a 


If instead, U;(x) = (+4) (x172 — 1) for each i, where 3; > 0 for each i, and a > 0 is the same 
for all users, then an optimal allocation is called a-fair, and the optimality condition for a-fairness 
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Dalia) <0 for all € A. 
: (x7) 


The limiting case œa = 1 corresponds to proportional fairness. The case a = O corresponds maxi- 
mizing a weighted sum of the rates. A limiting form of a fairness as a — oo is max-min fairness, 
discussed next. 
Max-Min Fair Allocation 

In the remainder of this section we discuss max min fairness, an alternative to the notion of 
proportional fairness. Perhaps the easiest way to define the max-min fair allocation is to give a 
recipe for finding it. Suppose that A is a closed, convex subset of R} containing the origin. Let e, 
denote the vector in R” with rt? coordinate equal to one, and other coordinates zero. Given x € A, 
let gr(x) = 1 if x + cer € A for sufficiently small e > 0, and g,(x) = 0 otherwise. Equivalently, 
g(x) is the maximal vector with coordinates in {0,1} such that x + eg(x) € A for sufficiently 
small e > 0. Construct the sequence (x) : k > 0) as follows. Let 2 = 0. Given 2), let 
glktl) = glk) + arg(a)), where az is taken to be as large as possible subject to r+!) € A. For 
some K < n, g(x*)) = 0. Then 49 is the max-min fair allocation. We denote it by a", 

Equivalently, if the coordinates of x 
nondecreasing order, then the reordered version of x 
reordered version of 2’. 


max-min and any other x € A are both reordered to be in 


maz=min is lexicographically larger than the 


Another characterization is that for any x € A, if 2, > gimar—min for some r, then there exists 
r so that Me E dl 

Still another characterization can be given for "0% in the special case that A = {x > 0: 
Ax < Ch, as in the congestion control problem with hard link constraints. Setting fM4e=mn = 


Agar=min the characterization is as follows: For each route r there is a link j, so that A = 


Cj and gmar—min — max{amarmin . j e r'}. Link jr is called a bottleneck link for route r. In 
words, a bottleneck link for route r must be saturated, and no other user using link jy can have a 
strictly larger allocation than user r. Due to this last characterization, a max-min fair allocation 
is sometimes called the envy free allocation, because for each user r there is a bottleneck link, and 
user r would not get a larger allocation by trading allocations with any of the other users using 
that link. 


8.9 A Network Evacuation Problem 


8.10 Braess Paradox 


Consider the network shown in Fig. 8.5. The arrival process is assumed to be a Poisson process of 
rate A. The upper branch of the network has a queue with a single exponential server followed by 
an infinite server queue with mean service time 2. The lower branch is similar, but with the order 
of the stations reversed. The traffic enters at a branch point, at which each customer is routed 
to one of the two branches, chosen independently and with equal probabilities. The scenario is 
equivalent to having a large number of independent Poisson arrival streams with sum of arrival 
rates equal to 2A, in which all customers from some streams are routed to the upper branch, and 
all customers from the other streams are routed to the lower branch. For reasons that are apparent 
below, assume that 0 < A < u— 1 < 2X. For example, A = 1.0 and y = 2.5. 
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Figure 8.5: A network with two routes. 
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Figure 8.6: Augmented network. 


Suppose that the choice of routing for each stream is made in a greedy fashion, so that no stream 
can improve its mean sojourn time by switching its routing choice. In game theory terminology, this 
is to say that the routing choices for the substreams form a Nash (or “person-by-person optimal”) 
equilibrium. By symmetry and by the convexity of the mean number of customers in an M/M/1 
queue as a function of arrival rate, the Nash equilibrium is achieved when half the customers are 
routed to the upper branch, and half to the lower branch. This leads to Poisson arrival processes 
with parameter A for each branch, and hence mean system time D = 2+1/(u— A) for both branches. 
The assumption that A < u — 1 implies that D < 3. 


Consider next the network of Fig. 8.6, which is obtained by augmenting the original network 
by a link with an infinite server queue with mean service time one. Let us consider what the new 
Nash equilibrium routing selection is. The augmented network has three routes: upper branch, 
lower branch, and cross-over route. Seek a solution in which all three routes are used by a positive 
fraction of the customers. This entails that the mean sojourn time for each of the three routes is the 
same. Let A denote the sum of the rates for the upper branch and cross-over routes. Equating the 
mean sojourn time of the cross-over route to that of the lower branch route yields that the mean 
system time, 1/(u — A”), of the single server station on the upper branch, should be 1. We thus set 
A = w—1. Our assumptions on à and y insure that A € (0,2A), as desired. The mean sojourn 
time on the upper branch and lower branch should also be the same, so that the mean time in the 
single server system on the lower branch should also be one, so that its throughput should also be 
A. This determines the rates for all three routes, and the resulting traffic rates on the links of the 
network are shown in the figure. Further thought reveals this to be the unique Nash equilibrium 
routing selection. 


Let us compare the mean sojourn time of customers in the two networks. The mean sojourn 
time in the original network is less than three, whereas in the augmented network it is equal to 
three. Thus, paradoxically, the mean sojourn time of all substreams is larger in the augmented 
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network than in the original network. Roughly speaking, the added link encourages customers to 
make heavier use of the single server systems, which are not as efficient from a social point of 
view. Of course if all streams of customers were to be banned from using the crossover link, then 
the performance observed in the original network would be obtained. This point corresponds to a 
Pareto optimal equilibrium point, and it clearly requires cooperation among the substreams. 


8.11 Further reading and notes 


The flow deviation algorithm was introduced in [15] and is discussed in [27], and is a special case 
of the Frank-Wolfe method. The flow deviation algorithms and other algorithms, based on use of 
second derivative information and projection, for congestion control and routing, are covered in [2]. 
For the decomposition of system problems into user and network problems, we follow [24]. Much 
additional material on the primal and dual algorithms, and second order convergence analysis based 
on the central limit theorem, is given in [25]. The notion of a-fairness is given in [31]. Additional 
material on algorithms and convergence analysis can be found in [38]. Braess discovered his paradox 
in a transportation network setting in 1968. It was adapted to the queueing context by Cohen and 
Kelly [6]. 

Today many subnetworks of the Internet use shortest path routing algorithms, but typically the 
routing is not dynamic, in that the link weights do not depend on congestion levels. Perhaps this 
is because the performance is deemed adequate, because there is plenty of capacity to spare. But 
it is also because of possible instabilities due to time lags. The recent surge in the use of wireless 
networks for high data rates has increased the use of dynamic routing (or load balancing) methods. 
Congestion control is central to the stability of the Internet, embodied by the transport control 
protocol (TCP) and van Jacobson’s slow start method. The decomposition method discussed in 
this chapter matches well the congestion control mechanism used on the Internet today, and it gives 
insight into how performance could be improved. 


8.12 Problems 


8.1. A flow deviation problem (Frank-Wolfe method) 
Consider the communication network with 24 links indicated in Figure 8.7. Each undirected 


E TOS 
@—@)—@) 


Figure 8.7: Mesh network with one source and one destination 


link in the figure represents two directed links, one in each direction. There are four users: 
S = {(1,9), (3,7), (9,1), (7,3)}, each with demand b. The initial routing is deterministic, being 
concentrated on the four paths (1, 2,5,6,9),(3,6,5,8,7),(9,8,5,4,1), and (7,4,5,2,3). Suppose 
that the cost associated with any one of the 24 links is given by D;(f) = fF /2, where f; is the total 
flow on the link, measured in units of traffic per second. 
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(a) What is the cost associated with the initial routing? 

(b) Describe the flow (for all four users) after one iteration of the flow deviation algorithm. Assume 
that, for any given user, any route from the origin to the destination can be used. Is the resulting 
flow optimal? 


8.2. A simple routing problem in a queueing network 
Consider the open queueing network with two customer classes shown in Figure 8.8. Customers of 


Source a 


NL) 
Source b ze, exit 


1-p~ LO— (2)— 


Figure 8.8: Open network with two types of customers 


each type arrive according to a Poisson process with rate one. All three stations have a single expo- 
nential server with rate two, all service times are independent, and the service order is first-come, 
first-served. Each arrival of a customer of type b is routed to the upper branch with probability p, 
independently of the history of the system up to the time of arrival. (a) Find expressions for Da(p) 
and Dy(p), the mean time in the network for type a and type b customers, respectively. (b) Sketch 
the set of possible operating points {(Da(p), Dilp)) : 0 <p < 1). (c) Find the value pa of p which 
minimizes Da(p). (d) Find the value p, of p which minimizes Di(p). (e) Find the value pave Of p 
which minimizes the average delay, (Da(p) + Dy(p))/2. Also, how many iterations are required for 
the flow deviation algorithm (Frank-Wolfe method) to find pave? (f) Find the value pm of p which 
minimizes max[D¿(p), Dy(p)}. 


8.3. A joint routing and congestion control problem 


Consider the network shown in Figure 8.9 and suppose (1,4) and (2,3) are the only two origin- 


1) b+c 3 


f a 4\ 
2 a+d CC) 


Figure 8.9: Network with joint routing and congestion control 


destination pairs to be supported by the network. Suppose that the system cost for flow F on any 
link lis D)(F’) = a and that the utility of flow r;; for origin-destination pair (i, j) is 3); log(ri;) 
for constants 614, 323 > 0. Suppose that for each of the two origin-destination pairs that flow can 
be split between two paths. For convenience of notation we use the letters a,b,c,d to denote the 
values of the four path flows: a = £124, b = 1134, € = £213, and d = 2243. The joint routing and 
congestion control problem is to minimize the sum of link costs minus the sum of the utilities: 


1 
min S 5 (a +b? + +d + (b+c)? + (a+c)) — Bralog(a + b) — Baz log(c + d) 


(a) Write out the optimality conditions. Include the possibility that some flow values may be 
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zero. (b) Show that if all flow values are positive, then a = b and c = d. (c) Find the optimal 
flows if B14 = 66 and (3 = 130. Verify that for each route used, the price of the route (sum of 
D'(F;) along the route) is equal to the marginal utility of flow for the origin-destination of the route. 


8.4. Joint routing and congestion control with hard link constraints 
Consider the joint routing and congestion control problem for two users, each of which can split 
flow over two routes, as shown in the figure. 


The total flow of user one is zı = a+ b and the total flow of user 2 is x2 = c+ d. The three 
central links each have capacity 8, and all other links have much larger capacity. User 1 derives 
value U¡(a + b) = 2va + b and user 2 derives value U2(c + d) = In(c + d). Let variables p1, p2, p3 
denote prices for the three links (i.e. Lagrange multipliers for the capacity constraints). 

(a) Write down the optimality conditions for a, b,c, d, pj, pa, p3- 

(b) Identify the optimal assignment (a,b,c,d) and the corresponding three link prices. 


8.5. Sufficiency of the optimality condition for hard link constraints 
Consider a network with link set L, users indexed by a set of fixed routes R, and hard capacity 
constraints. Use the following notation: 


e A= (A,,:1¢€ L,r € R) with Ar = 1 if link l is in route r, and Ar = 0 otherwise 
e x = (zr: r € R) is the vector of flows on routes 


= (fı: 1 € L) is the vector of flows on links, given by f = Ax 


f 
e C = (Cı: LE L) is the vector of link capacities, assumed finite 


U,(a,) is the utility function for route r, assumed concave, continuously differentiable, non- 
decreasing on [0, +00). 


(For simplicity routing is not considered.) The system optimization problem is maxz:2>0,Ar<C >_>, Ur(Ur). 
Suppose x* is feasible (1* > 0 and Ax* < C) and satisfies the following condition (with f* = Ax*: 
There exists a vector p = (pı) for each link such that 
pı = 0, for l € L, with equality if ff < Cı 
U'(xž) < Sp for r € R, with equality if af >0 


In this problem you are to prove that x* is a solution to the system optimization problem. 
(a) Define the Lagrangian 


L(x,p) = y U, (xt) + Sp (a > y a) 
r l 


riler 
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Show that L(1*,p) > L(x,p) for any other vector x with x > 0. (Hint: There is no capacity 
constraint, and L(x,p) is concave in x). (b) Deduce from (a) that x* is a solution to the sys- 
tem optimization problem. (Note: Since the feasible set is compact and the objective function is 
continuous, a maximizer x* exits. Since the objective function is continuously differentiable and 
concave and the constraints linear, a price vector p satsisfying the above conditions also exists, by 
a standard result in nonlinear optimization theory based on Farkas’ lemma.) 


8.6. Fair flow allocation with hard constrained links 
Consider four flows in a network of three links as shown in Figure 8.10. Assume the capacity of 


Flow 1 Link 1 Link 2 Link 3 


Flow 2 Flow 4 
Flow 3 


Figure 8.10: Network with four flows 


each link is one. (a) Find the max-min fair allocation of flows. (b) Find the proportionally fair 
allocation of flows, assuming the flows are equally weighted. This allocation maximizes >>, log(x;), 
where zx; is the rate of the ith flow. Indicate the corresponding link prices. 
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Chapter 9 


Dynamic Network Control 


9.1 Dynamic programming 


The following game illustrates the concept of dynamic programming. Initially there are ten spots 
on a blackboard and a pair of ordinary dice is available. The player, a student volunteer from the 
audience, rolls the dice. Let ¿ denote the number rolled on the first die and 7 denote the number 
rolled on the second die. The player then must either erase i spots, or j spots from the board. 
Play continues in a similar way. The player wins if at some time the number rolled on one of the 
dice is equal to the number of spots remaining on the board, so that all spots can be erased from 
the board. The player loses if at some time both numbers rolled are greater than the number of 
spots remaining on the board. For example, suppose the player begins by rolling (4,5) and elects 
to erase 4 and leave 6 spots on the board. Suppose the player next rolls (5,2) and elects to erase 2 
more and leave 4 spots on the board. Suppose the player next rolls (4,2). The player wins because 
the four spots can be erased, leaving zero behind. 

What is the optimal strategy for this game? What is the probability of winning for the optimal 
strategy? These questions are easily answered using dynamic programming. Fundamentally, dy- 
namic programming is about working backwards from the end of a game. In the example at hand, 
the strategy is trivial if there is only one spot remaining on the board. Indeed, given one spot 
remains, the player wins if and only if one or both of the numbers rolled on the dice is 1, which has 
probability wı = >: The player has no choice. If there are only two spots remaining, the player 
wins if one of the numbers rolled is a two (happens with probability in). If one of the numbers is a 
one but neither is a two (happens with probability 35), then the player erases one spot and will go 
on to win with probability eat because there will be one spot left. Thus, given the player has two 
spots left, the probabilty of winning the game is wa = E + + * H. Again, there is essentially no 
strategy needed. What should the player do if there are three spots left and the player rolls (1,2)? 
Since wa > wy, clearly it is optimal for the player to erase just one spot, leaving two on the board. 

In general. let w,, denote the probability of winning, for the optimal strategy, given there are 
n spots left on the board. Let wo = 1 and wz = 0 for k < 0. If there are n spots left and (7,7) 
is rolled, the player has the choice of leaving n — i spots or n — j spots remaining on the board. 
Clearly the optimal strategy is to make the choice to maximize the probability of winning. This 


leads to the equation, 
6 


6 
1 
We = 36 So y max{Wk—i, Wk—j} (9.1) 


i=1 j=1 
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Table 9.1: Probability of winning given n spots remaining 


Wn 
0.305556 
0.381944 
0.460455 
0.537375 
0.607943 
0.666299 
0.564335 
0.588925 
0.606042 
0.617169 


WOON DOKBWNHEH]S 


E 
© 


which recursively determines all the w,,'s. Numerical results are shown in Table 9.1. 

In particular, given ten spots initially, the probability of winning the game is 0.617169. To 
compactly describe the optimal strategy, we list the numbers 1 through 9 according to decreasing 
values of the w,'s: 6,5,9,8,7,4,3,2,1. Each time the player is faced with a decision, the optimal 
choice is the one that makes the number of spots remaining on the board appear earliest on the 
list. For example, if the first roll is (4,5), the player should erase 4 spots, because 6 appears on the 
list before 5. 

Note that the optimal policy is deterministic and that decisions are based only on the remaining 
number of spots. There is no advantage for the player to randomize decisions or to recall the past 
history of the game. Note also that while the dynamic program equation (9.1) was derived by 
conditioning on the outcomes of one step forward in time, the equation is solved essentially by 
working backward from the end of the game. Working backwards from the end is the essence of 
dynamic programming. 


9.2 Dynamic Programming Formulation 


Dynamic control with complete state information is studied using the dynamic programming ap- 
proach. The setting is pure jump controlled semi-Markov processes with independent and identically 
distributed holding times. The following are assumed given: 


e A finite or countably infinite state space S, 

e A real-valued function g on S, denoting instantaneous cost, 

e An action space U 

e A transition probability matrix P(u) = (pi;(u))ijes for each u € U. 

e A probability distribution function F', giving the distribution of inter-event times. 


Definition 9.2.1 A random process (X(t) : t > 0) is a controlled semi-Markov process with control 
policy m = (Wo, w1, W2,...) if 
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(a) (X(t) :t> 0) has pure-jump sample paths. 


(b) There is a sequence of random variables to = 0, t1,t2,... such that (tr +1 — tk)k>o are indepen- 
dent with common distribution function F. The jump times of (X(t) : t > 0) are all included 
in [to, t1, oe zk 


(c) wp is a function of (X (t;):0 < j < k) for each k 
(d) P[X(tk+1) = j | X (tk) = i, wk = u, (X (8) : s < te)] = pij(u) 


(e) (X (tx) : k > 0) is independent of (tp : k > 0) 


Notes: Condition (c) is a causality condition on the control policy m, while condition (d) is a 
“semi-Markov” property. Conditions (a)-(e) can be greatly relaxed in what follows. 

A cost function is used to evaluate a particular control policy m. A possible cost function is the 
discounted average cost 


T 
E, I (XE) dt, 


where e~ is a discount factor for inflation rate a > 0, T € (0,00] is the time horizon, and “E,” 


denotes expectation for initial state x(0) = x. Another cost function is the long term average cost: 


1 Efe 
lim supp oo Ex f g(X(t))at 
0 


In order to apply the dynamic programming approach in a simple setting, it is useful to transfer 
attention to an equivalent discrete time control problem. The main idea is that 


m -at -f E Ex ho B*g(X(tk)) ifa>0 
E, f A ae eS PKG). iba =0 


where 3 = F(a) = Le °*dF(s). Thus, control of the discrete time process (X(t;)) is equivalent 
to control of the original process (X(t)). Often we will write X(k) for X(t,). 

Example (Flow control — an Meontrotiea/M/1 queue) Consider a system consisting of a queue 
and a server. Let X(t) denote the number of customers in the system. Suppose customers arrive at 
instantaneous rate Au, where u denotes a control variable with u € [0,1]. Thus, A is the maximum 
possible arrival rate, and the effect of the control is to select a possibly smaller actual arrival rate. 
The server is an exponential type server with rate u. Thus, the parameters of the system are the 
maximum arrival rate A, and the service rate u. By taking the control u to be a constant, the model 
reduces to an M/M/1 system with arrival rate Au. In this example the control will be allowed to 
depend on the number in the system. 

Roughly speaking, the control strategy should insure a large throughput, while keeping the 
number in the system small. Of course, setting u = 1 maximizes the expected throughput, but, 
especially if A > pu, this can lead to a large number in the system. On the other hand, if u = 0 then 
the number in the system is minimized, but the long run throughput is zero. We will thus select a 
cost function which reflects both customer holding cost and a reward for throughput. 
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This example can be fit into the dynamic programming framework by choosing S = {0,1,2,...}, 
u=/0,1], 


| pean Au 0 0 | 
u A(1—u) Au 
Y Y Y 0 
Puy=| 0 Ma 
0 0 ` 


and taking F to be the exponential distribution function with parameter y = A+ p. 
A reasonable objective is to minimize the discounted average cost, 


cEs [ eX (t)dt — E, S Trax y=ye *, (9.2) 


t<T 


where a > 0 is a discount factor, and c > 0 is a constant. The integral term in (9.2) reflects holding 
cost c per unit time per customer in the system, and the sum term in (9.2) reflects a unit reward 
per departure. Replacing the departure process by its compensator (integrated intensity), the cost 
in (9.2) can be reexpressed as 


Be fe Totg A) phe fe -tdt (9.3) 
policy independent 


where g(x) = cx + ul (z=0)- Since the second term in (9.3) does not depend on the control policy 
used, the term can be dropped from the cost function. Thus, the function g taken in the dynamic 
programming formulation is g(x) = cx + wl{x = 0}. Intuitively, the cost g(0) = u accounts for lost 
departure opportunities when the system is empty, whereas the linear term cx gives the holding 
cost of customers in the system. 


9.3 The Dynamic Programming Optimality Equations 


The discrete time problem is to choose 7 to minimize E, ay Brg(X(k)). 
Define 


n—1 
Valz) = inf Es Y 8 g(X(k 
k= 


The value V,(x) is the minimum cost if n terms are included in the cost function, given initial 
state x. Set by convention Vo(x) = 0, and observe that Vi (x) = g(x). The fundamental backwards 
equation of dynamic programming is 


Vata (ar) = g(a) + 6 inf > | pey(u)Va(y); (9.4) 
y 


for n > 0. Furthermore, a policy 7 is optimal if the control value when there are n steps to go is a 
function u, (1) of the present state, where the feedback control function un is given by 


un (2) = argminucy Y Poylu) Val) (9.5) 
y 
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Together (9.4) and (9.5) determine the optimal control values. If the state space is finite, these 
equations provide a numerical method for computation of the optimal policies. For very large or 
infinite state spaces, some progress can often be made by approximating the process by one with a 
moderate number of states and again seeking numerical results. Still another use of the equations 
is to verify structural properties of optimal feedback control functions u,, (such as a “threshold” or 
“switching curve” structure). These properties can serve to restrict the search for optimal policies 
to a much smaller set of policies. An approach to verifying structural properties for the u,, is to 
seek a set of properties for the cost-to-go functions Vp: (a) which can be proved by induction on n 
via (9.4), and (b) which imply the structural properties of the u, via (9.5). 

Return to Example To illustrate the method, let us return to the Meontrotea/M/1 example , 
and prove the following claim: An optimal decision for n steps to go is given by 


TE os 


for some threshold value s(n). 
The dynamic programming recursion (9.4) for this example becomes 


Vasile) = co + plazo) +A min (Evae D) a) ne) (97) 
and (9.5) becomes 
0 if Va(£) < Valz +1) 
üna) =< 1 if V, (x) > Valz +1) (9.8) 


arbitrary if Va(x) = Va(x +1) 
Equation (9.8) shows that without loss of optimality, the control values can be restricted to values 
0 and 1. Furthermore, (9.7) simplifies to 


Vasile) = cx + plazo) + vo ai) 4 min(V,(2), Vale +1) (9.9) 


In view of (9.8), the claim is implied by the following property of the Vp: 


<0 ifa< s(n) 


>0 if x> s(n) (a10) 


Val + 1) — Vn (2) { 
for some threshold value s(n). While (9.10) implies that the structure (9.6) is optimal, the property 
(9.10) does not follow by induction from (9.9). Thus, a stronger property is sought for V, which 
can be proved by induction. 

The following two properties can be proved jointly by induction on n: 


(a) V;,(a) is convex (i.e. V,(a + 1) — V,,(x) is nondecreasing in 2) 
(b) Vn(1) — Vn(0) 2-9/8. 


Also, property (a) implies (9.10) for some value of the threshold s(n), and hence establishes that 
the threshold structure for un is optimal. 

Proof. Here is the induction argument. The base case, that Vo satisfies (a) and (b), is imme- 
diate. Suppose that Vn satisfies properties (a) and (b). It must be shown that V,+1 has properties 
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(a) and (b). Define V,,(—1) = V,(0) + (y/8). By property (b) for Vn, the new function Vp is convex 
on the set {—1,0,1,...}, and (9.9) becomes 


ala = cn + BV, (G 1).)4 A A Vek: (9.11) 


for x > 0. Since each of the three terms on the right of (9.11) is convex on {0,1,2,...} (sketch an 
example for the last of the three terms) it follows that V,,,1 satisfies property (a). E 


Note that property (b) of V, was used to prove property (a) for V,, +1. In fact, that is the only 
reason we chose to introduce property (b). It remains to verify property (b) for V,41. That is 
accomplished by the following string of inequalities, where “a A b” denotes the minimum of two 
numbers a and b: 


Vas1(1) — Vas (0) = —p+e+ Pe) A Va(1)) — (Va (1) A Vn(0))] 
i 5 Wa) A Vn(1)) — Va(1)] 
= -u+ PA vn) —Vn(1)) A0 
em Wall) — V,(0)) A0 
> Sp = = 1200 


The proof that V,,+1, and hence all the functions Vp, have properties (a) and (b), is proved. 

Some remarks concerning extensions of the above result are in order. The threshold values s(n) 
depend on n. However, it can be shown that limp... V, (1) exists and is finite for each x. Call the 
limit function V(x). It can also be shown that the limit Vo determines the optimal strategies 
for the infinite horizon, discounted cost problem. Since properties (a) and (b) are inherited by the 
limit, conclude that there is an optimal control strategy for the infinite horizon problem given by 
a fixed threshold s: new arrivals are allowed if and only if the number in the system is less than 
s. Moreover, unless it happens that Vo (1) = Voo(a +1) for some value of x, the threshold policy 
is the unique optimal control. It can also be shown that limg_,1 [Vo (1) — Voo(0)] exists and gives 
rise to the control for the long-term average cost criterion. The limit is again convex, so that the 
long-term average cost is also minimized by a fixed threshold policy. 

Example (Dynamic routing to two exponential server queues) 

Consider a system with two service stations, each consisting of a queue plus server, as shown 
in Figure 9.1. Customers arrive according to a Poisson process of rate A > 0. A customer is routed 
to the first station with probability u, and to the second station with probability 1 — u, where u is 
a variable to be controlled, and can depend on the state of the system. Service times at the first 
station are exponentially distributed with parameter u1, and service times at the second station 
are exponentially distributed with paramenter u2. The holding cost per unit time per customer in 
station 1 is cı and in station 2 is c2, where c; > 0 for i = 1,2. This example fits into the dynamic 
programming framework we have described, with the following choices: 


U = (0,1) 
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S= z2, where state x = (11,12) denotes x; customers in station i, for i = 1,2. 
g(x) =c101 + c2%2 


F corresponds to the exponential distribution with parameter y = A+ 1 + u2. 


Pay lu) = > (M1 Ly=D,0) + M2Ly=D30) + UAL {y= Ay2} + (1 — 4) AL (y=420)) Where 
Ají = (a1 + 1,22), Agr = (21,22 +1), Dix = ((21 — 1)4, £2), and Dox = (x1, (x2 — 1)+). 


The backwards equation of dynamic programming becomes 


Vn+1(£) = g(x) + ae : [p1V,(D¡2) + pa V, (Dax) + udrAV, (Aix) + (1 — u) AV, (A22)) 


or, after plugging in the optimal value for u, 
Vn4i(x) = g(x) + : {41 Vn (D12) + 2 Va (Dax) + Amin[V,(A12), Vn(A2x)),) 


with the intial condition Vj = 0. Furthermore, the optimal control is given in feedback form as 
follows. Given there are n steps to go and the state is x, the optimal control action is už (x), given 
by 

i < Va(A 
* (x) = { 1 if V,(A12) S Val 21) (9.12) 


0 else. 


That is, if the current state is x and an arrival occurs, the optimal control routes the arrival to 
whichever station yields the lower expected cost. 

Consider the symmetric case: uy = u2 and cı = c2. We will prove that u; (1) = Ma for 
all n > 1. That is, the send to shorter policy is optimal. It is easy to check that the control 
specification (10.20) is equivalent to u% (x) = Itri <z2}, if Vn has the following three properties: 

1. (symmetry) V (z1, 12) = V (z2, 21) for all (11,12) E S 

2. (monotonicity) V (11,12) is nondecreasing in zı and in z2. 

3. V(a1, 22) < V(x, — 1, £2 + 1) whenever 0 < 21 < zo. 

Together properties 1 and 3 mean that when restricted to the states along a line segment of the 
form {x E€ S : xı + £2 = l} for l > 1, the cost function is a monotonically nondecreasing function 
of the distance from the midpoint. 

Thus, it remains to prove that V,, has the three properties stated, for all n > 0. This is done 
by induction. Trivially, the function Vo satisfies all three properties. So for the sake of argument 
by induction, suppose that V, satisfies all three properties. We show that V,+1 satisfies all three 


station 1. == 
x Ze: 


station 2 ae 


Figure 9.1: Dynamic routing to two service stations 
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properties. Property 1 for V,, +1 follows from property 1 for Vp and the symmetry of the backwards 
equations in zı and x2. Property 2 for V,.+1 follows from property 2 for Vn. It remains to show 
that Vn+ı has property 3. Since property 3 is closed under summation, it suffices to prove that 
g(x), Va(Dıx), Va(D2x), and mint Vp (411), Vn(A2x)} have property 3. It is easily checked that the 
first three of these functions has property 3, so it remains to prove that min{V,,(A12), V,,(421)) 
has property 3. So suppose that 1 < zı < 29, and refer to Figure 9.2. It must be shown that 


ec 


O 
(x-1,x,+D 


xO 
e 


Figure 9.2: V, is sampled to get values a, b, and c. 


aAb< bAc where a = V, (11 +1, 22), b= V,(11, £2 +1), and c= Va (11 — 1, £2 +2). We argue first 
that a < b, by considering two cases: If xı = x2 then the states (11,12 + 1) and (11 + 1,22) are 
obtained from each other by swapping coordinates, so that a = b by property 1 for Vn. If x1 > x 
then 11 > x2 +1 so that a < b by property 3 for Vn. Thus, in general a < b. Similarly, b < c by 
property 3 for Vn. Thus, a < b < c, which immediately implies a Ab < b ^ c, as desired. The proof 
of the induction step is complete. Therefore, Vp has properties 1-3 for all n > 0, and the send to 
shorter queue policy is optimal, in the symmetric case. 

In the general case, the optimal control is given by už (x) = I {a2>sn(v1)}1 Where sn is a nonde- 
creasing function. This fact can be established by proving, by induction on n, that the cost-to-go 
functions V, have a more elaborate set of properties [19]. 


9.4 Problems 


9.1. Illustration of dynamic programming — a stopping time problem 

Let n > 1 and let Xy,..., Xn be mutually independent, exponentially distributed random variables 
with mean one. A player observes the values of the random variables one at a time. After each 
observation the player decides whether to stop making observations or to continue making observa- 
tions. The player’s score for the game is the last value observed. Let V,, denote the expected score 
for an optimal policy. Note that Vı = 1. (a) Express V,+1 as a function of Vp. (Hint: Condition on 
the value of X1.) (b) Describe the optimal policy for arbitrary value n. (c) For 1 < n < 5, compare 
V, to E[max(Xy,...,Xp)] =1+5+ i +---+4, which is the maximum expected score in case the 
player learns the values of the X’s before play begins. 


9.2. Comparison of four ways to share two servers 
Let N; denote the mean total number of customers in system 7 in equilibrium, where systems 1 


through 4 are shown. The arrival process for each system is Poisson with rate 2A and the servers 
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are exponential with parameter u. (a) Compute N; as a function of p = A/p for 1 < i < 3. (Let 


1. Za 
ie W= Q- 
u Y 
2 s OS x S 2D s Every other customer 
independent splitting goes to the tepqueng 
0.5 
mo- ~k- 
o 
3. 4 
2-server queue 
o= ay mO 
2) —_ / Arrivals sent to the shorter queue 


Z 
o  <IMO- 


Figure 9.3: Four two-server systems 


me know if you can find Ny. ) (b) Order Ni, Na, N3, N4 from smallest to largest. Justify answer. 
Does your answer depend on p? 


9.3. A stochastic dynamic congestion control problem 

Consider a single server queue with finite waiting room so that the system can hold at most K 
customers. The service times are independent and exponential with parameter u. If a customer 
arrives and finds the system full then the customer is rejected and a penalty c is assessed. For each 
customer that is served a reward r is received at the time of the customer's departure. There are 
two sources of customers. One is Poisson with fixed rate A, and the other is a variable rate source 
with instantaneous rate Au, where A is a constant and u is the control value, assumed to lie in the 
interval [0,1]. The control value u can depend on the state. 

(a) Formulate a dynamic programming problem. In particular, describe the state space, the in- 
stantaneous cost function, the distribution of inter-event times, and the transition matrix P(u) 
(draw the transition probability diagram). (b) Give the equations for the cost-to-go functions Vn. 
(Assume a discount factor 8 < 1 for the discrete time formulation as in class.) (c) Describe the 
optimal control law explicitly in terms of the cost-to-go functions. (d) Speculate about the struc- 
ture of optimal policy. Can you prove your conjecture? (e) Suppose the assumptions were modified 
so that customers from the first source are served at rate uı and that customers from the second 
are served at rate uo. Describe the state space that would be needed for a controlled Markov 
description of the system under (i) FCFS service order, (ii) preemptive resume priority to customer 
from the first source, or (iii) processor sharing. 


9.4. Conversion to discrete time with control dependent cost 
The continuous time control problem can still be converted to a discrete time control problem if 
the cost per unit time function g depends on u because, in that case, 


PE, ano PX (ty), We) if a > 0 


7 n ; 9.13 
ty Ez Y go 9(X (tk), Wr) ifa=0 ( ) 


tn 
By gx eeta 
0 
where 6 = F(a) = Len **aF(s). (a) Prove (9.13) in case a > 0. (The proof for a = 0 is sim- 
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ilar.) Thus, by ignoring constant factors, we can take the cost for n terms in discrete time to be 


Es Vico IX (te), wk). 


(b) How should the fundamental backwards recursion of dynamic programming be modified to take 
into account the dependence of g on u? 


9.5. Optimal control of a server 
Consider a system of two service stations as pictured. 


Ll station 1 ane 

u 
A DAS 
I station 2 


Station 7 has Possion arrivals at rate A; and an exponential type server, with rate m;(u), where 
mi(u) = fy + uuo and ma(u) = u2 + (1 — u) uo and u is a control variable with u € U = [0,1]. 
Suppose we are interested in the infinite horizon discounted average number of customers in the 
system, with discount rate a > 0. 

(a) Specify the transition probability matrix, P(u), the interjump CDF, F, and cost function g, for 
a dynamic programming model. 

(b) Write the dynamic programming update equations for the optimal cost-to-go functions, Vn. 
(c) Express the optimal state feedback control for n-steps, u% (x), in terms of Vn. 

(d) Consider the symmetric case: A¡ = A2 and fy = u2. Give an educated guess regarding the 
structure of the optimal control, and give an equivalent condition on V,. (Note: An equivalent 
condition, not just a sufficient condition, is asked for here.) 


9.6. A dynamic server rate control problem with switching costs 

Consider the following variation of an M/M/1 queueing system. Customers arrive according to 
a Poisson process with rate A. The server can be in one of two states—the low state or the high 
state. In addition, the server is either busy or idle, depending on whether there are customers in 
the system. A control policy is to be found which determines when the server should switch states, 
either from low to high or from high to low. The costs involved are a cost per unit time for each 
customer waiting in the system, a cost per unit time of having the server in the high state, and a 
switching cost, assessed each time that the server is switched from the low state to the high state. 
Specifically, let uy (respectively, uz) denote the service rate when the server is in the high state 
(respectively, low state), where uy > ur > 0. Let cy denote the added cost per unit time for 
having the server in the high state, and let cs denote the cost of switching the server from the low 
state to the high state. Assume that the control can switch the state of the server only just after 
a customer arrival or potential departure. Assume there is no charge for switching the server from 
the high state to the low state. Finally, let cy denote the cost per unit time per customer waiting 
in the system. The controller has knowledge of the server state and the number of customers in 
the system. Suppose the goal is to minimize the expected cost over the infinite horizon, discounted 
at rate a. (a) Formulate the control problem as a semi-Markov stochastic control problem. In 
particular, indicate the state space you use, and the transition probabilities. (Hint: Use a control 
dependent cost function to incorporate the switching cost. The switching cost is incurred at discrete 
times, in the same way the rewards are given at discrete times for the M¢ontrollea/M/1 example 
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in the notes, and can be handled similarly. Using a two dimensional control, one coordinate to 
be used in case of an arrival, and one coordinate to be used in case of a departure, is helpful.) 
(b) Let V, denote the cost-to-go function when there are n steps in the equivalent discrete-time 
control problem. Write down the dynamic programming equation expressing V,,+1 in terms of Vp 
and indicate how the optimal control for n steps to go depends on Vp. (c) Describe the qualitative 
behavior you expect for the optimal control for the infinite horizon problem. In particular, indicate 
a particular threshold or switching curve structure you expect the control to have. (d) Describe 
the optimal control policy when the following two conditions both hold: there is no switching cost 
(i.e. cg = 0), and cw(ua +a) > (ur + ley + cH). (Hint: Consider the expected cost incurred 
during the service of one customer.) 
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Chapter 10 


Solutions 


1.1. Poisson merger 

Suppose X; is a Poisson random variable with mean A; for i € {1,2} such that Xı and X are 
independent. Let X = Xı + X2, A=A1 + A2, and p; = A¡/A. Then for k > 0, 

F ] A N 7 . . 

PIX = k] = Xj- PIM = JPX = k- 4] = Vico GS = SEIS rl = 
Da where we use the fact that the sum in square brackets is the sum over the binomial distri- 
bution with parameters k and pı. Thus, the sum of two independent Poission random variables is 
a Poisson random variable. By induction this implies that the sum of any number of independent 


Poisson random variables is also Poisson. 

Now suppose that N; = (N;(t) : t > 0) is a Poisson process with rate A; for 1 < j < K, and 
that Nı,..., Nx are mutually independent. Given to < tı < +++ < tp, consider the following array 
of random variables: 


Ni(ti) — Ni(to)  Mi(t2)— Ni(t1) oe Ni(tp) — Ni(tp-1) 
Na(t1) — Nalto)  Na(t2) — Na(t1) en Na(tp) — No(tp-1) 

Me (10.1) 
Nk(t1) — Nk(to) Nx(t2) — Nx(tr) fer Nx (tp) — Nx (tp-1) 


The rows of the array are independent by assumption, and the variables within a row of the ar- 
ray are independent and Poisson distributed by Proposition 1.5.1, characterization (iii). Thus, 
the elements of the array are mutually independent and each is Poisson distributed. Let N(t) = 
Nı (t) +--+: + Nk(t) for all t > 1. Then the vector of random variables 


N(t)—N(to) N(te)—N(h) << Mty) — N(tp-a) (10.2) 


is obtained by summing the rows of the array. Therefore, the variables of the row are independent, 
and for any i the random variable N(t;+1) — N(t;) has the Poi(A(A(ti+1 — ti)) distribution, where 
à= Ay +---+AxK. Thus, N is a rate A Poisson process by Proposition 1.5.1, characterization (iii). 


1.2. Poisson splitting 
This is basically the first problem in reverse. Let X be Possion random variable, and let each 
of X individuals be independently assigned a type, with type i having probability p;,, for some 


161 


probability distribution pi,...,px. Let X; denote the number assigned type i. Then, 


' . , ; i +- +ik j 
P(X, = i1, Ao = i2,- XK = ikg) = P= ate big) (Te j. Joti 
11: toe St" LK: 
1 
j= tj: 


where A; = Ap;. Thus, independent splitting of a Poisson number of individuals yields that the 
number of each type i is Poisson, with mean A; = Ap; and they are independent of each other. 
Now suppose that N is a rate À process, and that N; is the process of type i points, given 
independent splitting of N with spilt distribution p1,...,pg. Proposition 1.5.1, characterization 
(iii). the random variables in (10.2) are independent, with the i*” having the Poi(A(A(ti+ı — ti)) 
distribution. Suppose each column of the array (10.1) is obtained by independent splitting of the 
corresponding variable in (10.2). Then by the splitting property of random variables, we get that 
all elements of the array (10.1) are independent, with the appropriate means. By Proposition 
1.5.1, characterization (iii), the 1% process N; is a rate Ap; random process for each i, and because 
of the independence of the rows of the array, the K processes N;,..., Ng are mutually independent. 


1.3. Poisson method for coupon collector’s problem 

(a) In view of the previous problem, the number of coupons of a given type i that arrive by time t has 
the Possion distribution with mean Lo and the numbers of arrivals of different types are independent. 
Thus, at least one type i coupon arrives with probability 1 — e~*/*, and p(k,t) = (1 — E 
(b) Therefore, p(k, kln k + ke) = (1 — $2)", which, by the hint, converges to e~® © as k — oo. 

(c) The increments of A over intervals of the form [k, k+1] are independent, Poi(1) random variables. 
Thus, the central limit theorem can be applied to A(t) for large t, yielding that for any constant D, 
limo P[A(t) > t+ DvVt] = Q(D), where Q is the complementary normal CDF. The problem has 
to do with deviations of size (c — c')k, which for any fixed D, grow to be larger than DVkInk + ke. 
Thus, with e = |c—c|, P[|A(klnk + ke) — (kInk + kc)| > ek] < Q(D) for k large enough, for any 
D. Thus, limz-=0 P[|A(k Ink + ke) — (klnk + ke)| > ek] = 0, which is equivalent to the required 
fact. 

(d) Let C; be the event that the collection is complete at time t. Condition on the value of A(t). 


p(k, t) = P(C;) = P(C)|A(t) > n)P(A(t) > n) + P(CHA(E) < n)P(A(t) < n) (10.3) 


First we bound from below the right side of (10.3). Since getting more coupons can only help com- 
plete a collection, P(C;|A(t) > n) > P(C A(t) = n) = d(k,n), and other terms on the righthand 
side of (10.3) are nonnegative. This yields the desired lower bound. 

Similarly, to bound above the right side of (10.3) we use P(C l A(t) > n) < 1 and P(C;|A(t) < n) < 
d(k,n). 

(e) Fix c. The first part of the inequality of part (c) yields that d(k,n) < p(k,t)/P[A(t) > n]. Sup- 
pose n = klnk + kc. Let d > c and take t = klnk + kd as k — oo. Then p(k,t) => e€ ° 
by part (b) and P[A(t) > n] — 1 Thus, limsupd(k,n) < e”. Since d is arbitrary with 
d > c, limsupd(k,n) < e7**. A similar argument shows that liminf d(k,n) > e~® °. There- 
fore, lim d(k,n) = e° © 
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1.4. The sum of a random number of random variables 


E[S] = Drs = n]P[N = n] = o +--+ Xn N =n]P[N =n] 
= Y Elx ++ Xn]P[N =n] = Y nXPIN =n] =XN 
Similarly, T T 
E(s?] = Ya + + Xp) IPIN =n] 


= Y [EX ++ Xn]? + Var(X1 +-+- + Xn)] PIN = n] 


= `> [(nX)? + nVar(X)] PIN = n] 
n=0 
= NYX)? + NVar(X) 
so that Var(S) = E[9?] — Es? = Var(N)(X)? + NVar(X). 
(b) By the same reasoning as in part (a), 
slu) = Efes] = > EUA ++X0] PIN = n] 


n=0 


Y) 9x,(u)"P[N = n] = Bx, (u)) 
n=0 


1.5. Mean hitting time for a simple Markov process 
(a) 
1 a 0.5 
-a 0.5 
Solve 7 = tP and re = 1 to get t = C a ay a x04 ay) for all a € [0,1] (unique). 
(b) A general way to solve this is to let h; = E[min{n > 0|X(n) = 3}|X(0) = i], for 0 < i < 3. 
Our goal is to find hg. Trivially, hg = 0. Derive equations for the other values by conditioning on 
ho= 1+hi 
the first step of the process: h; = 1+ y pijh; fori A 3. Or ha= 1+(1-—a)ho+ahz yielding 
ho= 1+(0.5)h; 


ho ¿+1 
hy |= 4 . Thus, ho = 4 + 1 is the required answer. 
ho 241 


1.6. A two station pipeline in continuous time 
(a) S = {00, 01, 10, 11} 
(b) 
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À 
A u1 
Wo 
=A 0 A 0 
pa —p2-A 0 A 
Cc — 
(pg 0 pa =mM1 0 
0 0 Ha ple 


n = 0.4. 

(e) Let 7 = min{t > 0: X(t) = 00}, and define hs = E[r|X(0) = s], for s € S. We wish to find 
hoo = 0 hoo 0 
hoi = LR H2hoo h Ahir h 3 

hy. Hata HatA  m+À Bor Tf \ = pa = H2 = 1.0 this yields OL = . Thus, 
hio=  +ho1 hio 4 
hi = - + hio hi1 5 


hy, = 5 is the required answer. 


1.7. Simple population growth models 


(a) 


(b) ro(t) = 0. The Kolmogorov forward equations are given, for k > 1, by 
Ot — (k — 1)Amp_1(t) — kàrg(t). 
Multiplying each side by A* and summing yields 


OP(z,t) Ome lt) y ~ k-1 > k-1 
Ft = A Alz S rg (t)(k — 1)z — 2) mp(t)kz 
k=1 k=2 k=1 
2 ¿A 
Oz 


That is, P(z,t) is a solution of the partial differential equation: 


OP(z,t) 
Ot 


OP(z,t) 


= Me? — 2) 5 
z 


with the initial condition P(z,0) = z. It is easy to check that the expression given in the problem 
statement is a solution. Although it wasn’t requested in the problem, here is a method to find 
the solution. The above PDE is a first order linear hyperbolic (i.e. wave type) equation which 
is well-posed and is readily solved by the method of characteristics. The idea is to rewrite the 
equation as [Z Nz? — z) 2Z)P(a, t) = 0, which means that P(z,t) has directional derivative zero 
in the direction (—A(z — 22), 1) in the (z,t) plane. 

(c) Write N; = E[N(t)]. We could use the expression for P(z,t) given and compute N; = 
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lz=0. Alternatively, divide each side of the PDE by z — 1. Then since an = = 0, we 


get 2 [Pena] = 2 oP at) Letting z — 1 then yields the differential equation dNe = AN;, so 


OP(z,t) 
Oz 


z—1 
that N; =e. : 
(d) By part (b), P(z,t) = mia =e “(las ye" titan (oe ae) 


for k > 1. That is, the population size at time t has the geometric distribution with parameter 
—Àt 
EN, 


(e) For the deterministic model, ny = 204! = 2% = e02), Thus, the growth rate for the de- 


terministic model is roughly exponential with exponent (In2)A = (0.693)A, which is considerably 
smaller than the exponent A for the random model. 


1.8. Equilibrium distribution of the jump chain 


: 0 ifi=j 
The choice of B insures that (—Bqiini)ies is a probability distribution. Since Py; = { Mi igi E A 
~ii 


we have for fixed j, 


q 
5S- Bqi;T;) Pi; = — Bayt (= ) 
iES tig dii 
= BY may = —Bryqy; 
iij 

Thus, —Bz;q is the equilibrium vector for P”. In other words, —BTiqii = Ti ; 
Another justification of the same result goes as RIRE. For the original Markov process, —q;; l is 
the mean holding time for each visit to state i. Since Ti represents the proportion of jumps of the 
process which land in state i, we must have m; x mi (— q)” 1 or equivalently, m7 x (—qu;) Tr? 


1.9. A simple Poisson process calculation 
Suppose 0< s<tand0<i<k. 


PIN(s) =4M(t) =k] = 


(oan (“8S a) (tout) 
i/t RL 
OR) 


That is, given N(t) = k, the conditional distribution of N(s) is binomial. This could have been 
deduced with no calculation, using the fact that given N(t) = k, the locations of the k points are 
uniformly and independently distributed on the interval [0, t]. 


1.10. An alternating renewal process 

(a) Sampled periods between light changes L are deterministic, length one, so the forward residual 
lifetime y is uniformly distributed over [0,1]. Thus, Pllight changes] = P[y < 0.5] = 0.5. 

(b) By symmetry, y is independent of what color is spotted first. If green is spotted first, 

w = 0 if y > 0.5 


y+0.5 ify < 05 so that E[W|green spotted] = made + 0.5)du = 2. Similarly, 


8 
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E|W|red spotted] = filu — 0.5)du = z. Thus, E[W] = 34 + 2 = 4. This make sense, be- 
cause the light is red with a one half when the light is reached, and given it is red when 
reached, the average wait is 0.5. 

(c) Pllight changes] = P[y < 0.5] = Pa Pd) dy = e 5 = 3 dy 

(d) 2 blocks/minute (The point here is to express the sneer is ins ee If the length of a block 
is specified, the speed could be given in meters/second or miles per hour, for example.) 


1.11. A simple question of periods 
(a) The period of state 4 is GCD{4,6,8, 10, 12,14,...} = GCD{4, 6} = 2. 
(b) The process is irreducible, so all states have the same period. So state 6 must also have period 2. 


1.12. A mean hitting time problem 


(a) 


TQ = 0 implies 7 = (2, 2, 2). 


(b) Clearly a; = 0. Condition on the first step. The initial holding time in state ¿ has mean E and 
ll J 
. k . Ve =; Sa 0 p ao 
th t stat th probability p?, = —“4. Th a q90 02 
e next state is j with probability pj; = 2 ws ( oe dt ec O = 


q22 
7 ; ao = 1 
Solving yields ( a ) = ( 15 ) ; 


(c) Clearly a2(t) = 0 for all t. 


ag(t+h) = ao(t)(1+ qoh) + a1(t)qioh + o(h) 
a(t +h) = ao(t)goih + ar(t)(1 + q11h) + o(h) 


Subtract a;(t) from each side and let h — 0 to yield (Se, 201) = (09,01) ( E a ) with the 
10 qu 


inital condition (ao(0), a1(0)) = (1,0). (Note: the matrix involved here is the Q matrix with the 
row and column for state 2 removed.) 


(d) Similarly, 


Bolt=h) = (1+ qooh)Go(t) + qo1hb1(t) + o(h) 
Bilt=h) = qohbolt) + (1 + q11h)B1(t)) + o(h) 


— 260 
Subtract (;(t)’s, divide by h and let h — 0 to get: ( EA ) = ( ee ) ( Bo ) with 


qio qu Br 
Cep) = (2) 


1.13. A birth-death process with periodic rates 
(a) Let 0 = dado. Then Sı = Mp0" (1 + dy which is finite if and only if 9 < 1. Thus, the 
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process is positive recurrent if and only if 0 < 1. (In case O < 1, man = 0” /S1 and many1 = 0” Fa. .) 
(b) Ag = Ap. In general, rg = TAz/A, so that r = ~ if and only if the arrival rates are all equal 


(corresponding to Poission arrivals, and then PASTA holds.) 
1.14. Markov model for a link with resets 
(a) Let S = {0,1,2,3}, where the state is the number of packets passed since the last reset. 
A A A 
u 


(b) By the PASTA property, the dropping probability is 73. We can find the equilibrium distribu- 
tion 7 by solving the equation TQ = 0. The balance equation for state 0 is Aro = (1 — To) so that 


To = e The balance equation for state i € {1,2} is Am,_1 = OST so that 7 = Tolaga) and 
T2 = Tol +> F )2. Finally, Amz = p73 so that 73 = Tol mys = = 5 ae The dropping probability is 
T3 = a (This formula for 73 can be deduced with virtually no calculation from the properties 


of merged Poisson processes. Fix a time t. Each event is a packet arrival with probability T and 
is a reset otherwise. The types of different events are independent. Finally, 73(t) is the probability 
that the last three events before time t were arrivals. The formula follows.) 


1.15. An unusual birth-death process 
a) Sa = Lo E = _o( =)” < +20 (because p > 0.5) so X is transient. 
Sp = Ep Hen = Y (EE) < +00 (b 0.5) so X is transient 
aQ = 0 is equivalent to Akak = Leaps for k > 0, which is easily checked. 
b) aQ =0 1 ivalent to A pg Qag+1 for k > 0, which i ily checked 
(c) pi = 1 and, for i > 1, PŽ =] -— p and pes) = p. All other transition probabilities are zero. 


CUE » 


1-p 


(d) S2 is the same as in (a), so X’ is transient (as is also obvious from part (c).) 
Note: X reaches the graveyard state A in finite time. Otherwise, the fact that Qa = 0 for a 
probability distribution a would imply that all states are positive recurrent. 


1.16. A queue with decreasing service rate 


(a) 


A A A A A A 
Cy GY 2. TRY mY ey 
Ch z% EA We Ses ae 

u u p u/2 u/2 u/2 


(b) Sp = DLE, where k A K = min{k, o Thus, if A < § then S2 < +00 and the 
process is recurrent. Sy = X p (2) 2 k^K soif A< # 5 then Sı < +00 a the process is positive 
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recurrent. In this case, mg = ri To, Where 


1 Qu, OME 17 
So 11-0/0 I=] 


(c) Eà s= e, the queue appears to be stable until if fluctuates above K. Eventually the queue- 


length will grow to infinity at rate A— 5 = §. See figure above. 


To = 


1.17. Limit of a distrete time queueing system 
(q) The transition rate diagram for the number in the system is shown: 


OLAS 


laq  1-aq-gp l-aq-gp 1-aq-gp 


ap = Pino arrivals for k — 1 slots, then arrival] = (1 — aq)*-taq for k > 1. Similarly, bi = 


(1 — 6q)*-'Gq for k > 1. Thus, an interarrival time is q times a geometric random variable 


with parameter aq, and the mean interarrival time is 1. and a service time is q times a geometric 


random variable with parameter Pq, and the mean service time is 3 


Q 


(b) Bqpk+1 = agpr for k > 0, impies that py = (1— p)p*, where p = 3- This distribution is the same 
for all q, so it trivially converges as q > 0 to the same value: py — (1— p)p*. The interarrival times 
and service times become exponentially distributed in the limit. The limit system is a continuous 
time M/M/1 system. 


1.18. An M/M/1 queue with impatient customers 


(a) 


a a » X $ 

¡CA Gs A AO E 

OBOEORONON 

u Uta u+2a u+3a u+4a 
(b) The process is positive recurrent for all À, u if a > 0, and pk = A where c is 
chosen so that the p;’s sum to one. 
k 

(c) If a = u, Pk = ee = $. Therefore, (px : k > 0) is the Poisson distribution with mean p. 


Furthermore, pp is the mean departure rate by defecting customers, divided by the mean arrival 
rate A. Thus, 

Cae de as p >00 

— 


¡Es 
po = x 2pk= Do = 0 asp>0 


where l'Hóspital's rule can be used to find the limit as p — 0. 
1.19. Statistical multiplexing 


(a) Use an M/M/1 model, with A = 40 packets/second, and y = 50 packets per second. Then 
T = ++ = 0.1 seconds. 


Au 
(b) Model each link as an M/M/1 queue, with A = 20 packets/second, and y = 25 packets per 
second. Then T = on = 0.2 seconds. 


(c) Model as an M/M/2 queue with A = 40 packets/second and u = 25 packets /second. 
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F A E A i A A 7 A 
OROBOROA ORL. 


ERA AA A 


u 2u 2u 2u 2u 


Let p = =. Then ro = Eo $ and mk = 2nop* for k > 1. Thus 
N oo k ie 2 
T= —= pan ae = elt p) = 0.111 seconds 
Aà AOS) A5) 0-5) 


We can view model (c) as a variation of (b) with statistical multiplexing at the packet level, and 
(a) as a variation with multiplexing at the subpacket level. Here (c) is a substantial improvement 
over (b), and (a) is a little better than (c). 


1.20. A queue with blocking 
(a) 


A A A 
fox YoY) GY Yay DA yY, A 
A 
UA QU A De oe IN pe J 
u u u u u 


pr _ p*(1=p) 
Tk = Gees epee T 1y for0<k<5. 
(b) pp = Ts by the PASTA property. 
(c) W = Nw/(A(1 — pg)) where Nw = N;_ o(k — 1)7;. Alternatively, W = N/(A(1 — pp)) — 7 
(i.e. W is equal to the mean time in system minus the mean time in service) 
= 1 1 
(d) ™ = \(mean cycle time for visits to state zero) — A(1/A+mean busy period duration) There- 


fore, the mean busy period duration is given by xl 1] = e a a 5) 


1.21. Multiplexing circuit and packet data streams 
(a) A portion of the rate diagram for C = 3 is pictured. 


e eJ f\ f) 
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(b) Yes, the phase structure is apparent. 


= pio le MEE A 
< = j= 
(c) Ap = (O no) = pee Aen JA 


°C 
1.22. Three queues and an autonomously traveling server 

(a) The equilibrium distribution for the server is m = (m1, T2, 13) = (27, Bz? A where Z = 
4 + 3 + al and the service capacity of station 7 is thus 7;, for 1 < 1 < 3. 

(b) The equilibrium distributions for the individual stations can be found one at a time, by the 
method of phases (even though the stations are not independent). For a fixed station 7, the state 
space would be [(1,0) : 1 > 0,1 < 0 < 3}, where / is the number of customers at station ¿and @ is 
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the phase of the server. 


1.23. On two distibutions seen by customers 


NO 


As can be seen in the picture, between any two transtions from state k to k+1 there is a transition 
form state k + 1 to k, and vice versa. Thus, the number of transitions of one type is within one of 
the number of transitions of the other type. This establishes that |D(k,t) — R(k,t)| < 1 for all k. 
(b)Observe that 


D(k,t) a a eS R(k,t) 


P en R(k,t) 


At Ôt Qt Qt Qt Or 
< 1 R(k, t) 1 Qt 
= Qt Qt Ôt 
1 At 
< + 11 >0 as t>00 
Ot Ot 


Thus, PLE) and i have the same limits, if the limits of either exists. 
2.1. Recurrence of mean zero random walks 

(Note: The boundedness condition is imposed to make the problem easier — it is not required 
for the results.) (a) Let V(x) = |x|, the absolute value of x. Then for x > M and any 
k > 0, |z + Bk| = « + By with probability one, by the boundedness of Bj. Thus, if « > M, 
PV (x) — V(x) = Ellx + Bel] — x = Elx + By] — x = 0. Similarly, if x£ < —M, PV (x) — V(x) = 
Ellx + By|] — |z| = Eļ|z| — Bx] — lx] =—E[Bx] = 0. Therefore, the Foster-Lyapunov criteria for 
recurrence are satisfied by P, V, and the set C = {x : |x| < M}, so that X is recurrent. 

(b) Let V(y) = y. As in part (a), we see that PV (y) — V (y) = 0 off the finite set C={y:0<y< 
M}, so that Y is recurrent. 


2.2. Positive recurrence of reflected random walk with negative drift 
Let V(x) = 527. Then 


(e+ Bn + En)" r? 

2 2 

(+ Br), z? 
2 


PV(x)- V(x) = El 


E 

EA 
— B2 

= aB4+= 
TD +- 


Therefore, the conditions of the combined Foster stability criteria and moment bound corollary 
apply, yielding that X is positive recurrent, and X < > (This bound is somewhat weaker than 
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Kingman's moment bound, disussed later in the notes: X < Vara) ) 


2.3. Routing with two arrival streams 

We begin by describing the system by stochastic evolution equations. For i € {0,1}, let B;(t) 
be a Bernoulli(a;) random variable, and let U;(t) be a Bernoulli(u;) random variable, and for 
j € {1,2,3} let D;(t) be a Bernoulli(d;) random variable. Suppose all these Bernoulli random 
variables are independent. Then the state process can be described by 


Xi(t+1) = X;(t) + As(t) — D¡(t) + Lit) for i € {1,2,3} (10.4) 
where 


A(t) = Ux(t)Bi(t) 
Ax(t) (1 — Ui (t)) 
A3(t) = (1—Ux(t)) 


By(t) + U2(t) Ba(t) 
Ba(t) 
and Lilt) = (—(Xi(t) + Ai(t) — Di(t))) +. 

Assume that the parameters satisfy the following three conditions: 


ajtag < dí +d2+d3 (10.5) 
ay < di+d2 (10.6) 
a < d2+ d3 (10.7) 


These conditions are necessary for any routing strategy to yield positive recurrence. Condition 
(10.5) states that the total arrival rate to the network is less than the sum of service rates in the 
network. Condition (10.6) states that the total arrival rate to servers 1 and 2 is less than the 
sum of the service rates of those servers. Condition (10.7) has a similar interpretation. We will 
show that conditions (10.5)-(10.7) are also sufficient for positive recurrence for random routing for 
a particular choice of u. To that end we consider the Lyapunov-Foster criteria. 

Letting V = (x? +x2+ x2) and arguing as in Example la, and using the fact (A;(t)— D:(t))? < 1 
for i = 1,3 and (A2(t) — Da(t))? < 4 yields (writing a for 1 — u;): 


3 
PV(z)-V(z) < (> r¡ElA¡(t) — DHX (t) = 3) +3 
i=1 
=  21(a1u1 — dı) + z2(a1U1] + a9u3 — da) + 23(aguz — d3) + 3 (10.8) 
< (21 + La + x3) minfd; — 0141, də — 4141 — 49343, d3 — aguz} +3 (10.9) 


To obtain the best bound possible, we select u = (u1, u2) to maximize the min term in (10.9). For 
any u, min{dı — a141, dz — a1U1 — agua, dz — aguzp, is the minimum of three numbers, which is less 
than or equal to the average of the three numbers, is less than or equal to the average of any two 
of the three numbers, and is less than or equal to any one of the three numbers. Therefore, for any 
choice of u € [0, 1]?, 

min{d, — a11, d2 — 414] — aque, d3 — aguz} < € (10.10) 
where 
dı + d2 + dz — aı — a2 dı + d2 — aı d2 + dz — a 

3 i 2 i 2 


2 dí, da, da) 


e = min{ 
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Note that e > 0 under the necessary conditions (10.5)-(10.7). Taking uj € [0,1] as large as possible 
subject to dı — aru} > e, and taking už € [0,1] as large as possible subject to d3 — azu} > e, yields 


the choice d 
ay ag 


* di =E 
uy = —— 
It is not hard to check that dz — ajuj — agus > e, so that equality holds in (10.10) for u = u*. 


This paragraph shows that the above choice of e and u* can be better understood by applying 
the max-flow min-cut theorem to the flow graph shown. 


2 TAE 
e 
E d 37 € 


In addition to the source node s and sink node t, there are two columns of nodes in the graph. 
Nodes vı and v2 correspond to the two arrival streams, and nodes q1,q2 and q3 correspond to the 
three queues. There are three stages of links in the graph. The capacity of a link (s,v;) in the 
first stage is a;, the capacities of the links in the second graph are very large, and the capacity of 
a link (q;,t) in the third stage is dj — e. The choice of e above is the largest possible so that (1) 
all links have nonnegative capacity, and (2) the capacity of any s — t cut is greater than or equal 
to a, + a2. Thus, the min flow max cut theorem insures that there exists an s — t flow with value 
a, + ag. Then, už can be taken to be the fraction of the flow into v; that continues to qi, for i = 1 
and i = 2. 

Under (10.5)-(10.7) and using u*, we have PV(x) — V(x) < —e(11 + £2 +23) +3. Thus, the 
system is positive recurrent, and Xi+X2+X3< 31 
(b) Suppose each arrival is routed to the shorter of the two queues it can go to. For simplicity, let 
such decisions be based on the queue lengths at the beginning of the slot. As far as simple bounds 
are concerned, there is no need, for example, to take into account the arrival in the top branch when 
making the decision on the bottom branch, although presumably this could improve performance. 
Also, it might make sense to break ties in favor of queues 1 and 3 over queue 2, but that is not 
very important either. We can denote this route-to-shorter policy by thinking of u in part (a) as 
a function of the state, u = 4% (x). Note that for given x, this choice of u minimizes the right 
hand side of (10.8). In particular, under the route-to-shorter policy, the right hand side of (10.8) 
is at least as small as its value for the fixed control u*. Therefore, writing P** for the one-step 
transition matrix for the route to shorther policy, P®°V(x) — V(x) < —e(£1 + z2 +£ +3) +3, and 
under the necessary condtions (10.5)-(10.7), the system is positive recurrent, and X1+X24+X3 < 2. 


2.4. An inadequacy of a linear potential function 
Suppose x is on the postive x2 axis (ie. 11 = 0 and x2 > 0). Then, given X(t) = x, during 
the slot, queue 1 will increase to 1 with probability a(1 — dı) = 0.42, and otherwise stay at zero. 


'By using the fact that X; = 0 if the arrival rate to queue i is zero, it can be shown that for u = u*, X1+X24+X3 < 
3, where el = min{ se ditde—a1 : da+da—az }. 
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Queue 2 will decrease by one with probability 0.4, and otherwise stay the same. Thus, the drift 
of V, ElV(X(t +1) — V(ax)|X(t) = z] is equal to 0.02. Therefore, the drift is strictly positive for 
infinitely many states, whereas the Foster-Lyapunov condition requires that the drift be negative 
off of a finite set C. So, the linear choice for V does not work for this example. 


2.5. Allocation of service 
Let e > 0. The largest we can make 1 (u) is mı and the largest we can make ju3(u) is m2. Thus, 
we can select u so that 


Arte<pi(u) and Az+e< uglu) (10.11) 


if and only if mi > A, + € and ma > Az + €. If we select u so that equality holds in (10.11) then the 
value of uə(u) is maximized subject to (10.11) , and it is given by 


ulu) = mi + ma — pı (u) — alu) = mi + ma — A + Az — 2e 


Therefore, in order that A2 + € < pa(u) hold in addition to (10.11), it is necessary and sufficient 
that e < Uttàs -mH and my > \y +e and ma > A» + e€. Thus, the value of e given in the 
example is indeed the largest possible. 


2.6. Opportunistic scheduling (Tassiulas and Ephremides [40]) 

(a) The left hand side of (2.27) is the arrival rate to the set of queues in s, and the righthand side 
is the probability that some queue in s is eligible for service in a given time slot. The condition is 
necessary for the stability of the set of queues in s. 

(b) Fix e > 0 so that for all s € E with s 4 0, 


Matos Y ws) 


ics B:Bns#0 


Consider the flow graph shown. 


In addition to the source node a and sink node b, there are two columns of nodes in the graph. The 
first column of nodes corresponds to the N queues, and the second column of nodes corresponds 
to the 2" subsets of E. There are three stages of links in the graph. The capacity of a link (a,q;) 
in the first stage is a; + €, there is a link (q,s;) in the second stage if and only if q; € sj, and each 
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such link has capacity greater than the sum of the capacities of all the links in the first stage, and 
the weight of a link (sz,t) in the third stage is w(sz). 

We claim that the minimum of the capacities of all a — b cuts is vx = pu ¡(a; + €). Here is a 
proof of the claim. The a — b cut ({a} : V — {a}) (here V is the set of nodes in the flow network) 
has capacity v*, so to prove the claim, it suffices to show that any other a — b cut has capacity 
greater than or equal to v*. Fix any a — b cut (A: B). Let A=ANn [q1,..., qn}, or in words, A 
is the set of nodes in the first column of the graph (i.e. set of queues) that are in A. If q € A and 
sj € B such that (q,s;) is a link in the flow graph, then the capacity of (A : B) is greater than or 
equal to the capacity of link (qi, sj), which is greater than v*, so the required inequality is proved 
in that case. Thus, we can suppose that A contains all the nodes s; in the second column such 
that s; N A #0. Therefore, 


C(A: B) > y (a; +e) + y w(s) 
¡€[q1,..,an }-A sC E:sn AZO 
> y (a; +€) + X (a; +e)=0v*, (10.12) 
¡€[q1,..,an }-A ¡EA 


where the inequality in (10.12) follows from the choice of e. The claim is proved. 

Therefore there is an a—b flow f which saturates all the links of the first stage of the flow graph. 

Let u(i, s) = F(q;, s)/f(s, b) for all i,s such that f(s,b) > 0. That is, u(i, s) is the fraction of flow 
on link (s, b) which comes from link (q;,s). For those s such that f(s,b) = 0, define u(i, s) in some 
arbitrary way, respecting the requirements u(i, s) > 0, u(i, s) = O if i ¢ s, and) ¡e y ult, s) = Lis 40)- 
Then a; + € = Ha, qi) = X, f(a. s) = do, f(s, bjuli, s) < DO, w(s)u(i, s) = ui (u), as required. 
(c) Let V(x) = 5 ¡en 27. Let 5(t) denote the identity of the queue given a potential service at 
time t, with 9(t) = 0 if no queue is given eat service. Then Pló(t) = i|S(t) = s] = u(i, s). The 
dynamics of queue i are given by X;(t +1) = X;(t) + As(t) — Ri(0(t)) + Li(t), where Ri(9) = Ips. 
Since Wyep(Ai(t) — Ri(6i(t)))? < Ejer)? + (RASO)? < N + Zier Alt)? we have 


PV(2)-Vía) < (x xila; — wt) +K (10.13) 
1€B 
E E B J +K (10.14) 
1€B 


where K = N + yd Ki. Thus, under the necessary stability conditions we have that under the 
vector of scheduling probabilities u, the system is positive recurrent, and 


x pes = (10.15) 


(d) If u could be selected as a function of the state, x, then the right hand side of (10.13) would 
be minimized by taking u(i,s) = 1 if i is the smallest index in s such that x; = maxjes xj. This 
suggests using the longest connected first (LCF) policy, in which the longest connected queue is 
served in each time slot. If P’©* denotes the one-step transition probability matrix for the LCF 
policy, then (10.13) holds for any u, if P is replaced by POF. Therefore, under the necessary 
condition and e as in part (b), (10.14) also holds with P replaced by PŁCF, and (10.15) holds for 


174 


the LCF policy. 


2.7. Routing to two queues — continuous time model 

Suppose À < 11 + u2, which is a necessary condition for positive recurrence. 

(a) Under this condition, we can find u so that uy > Au and u2 > AU. If each customer is indepen- 
dently routed to queue 1 with probability u, and if V(x) = (x? + 22)/2, (2.20) becomes 


2QV (2) = ((@1 + 1)? — 27) Au + ((£2 + 1)? — x3) NU + ((z1 — Dí — 27) pa + ((a2 — 13 - 23) 12. 


Since (11 — 1)? < (x; — 1)?, it follows, with y = A+ ua + pa, that 
QV (2) < — (x1(p1 — Ud) + mo (ue — TA)) + >. (10.16) 
Thus, the combined stability criteria and moment bound applies, yielding that the process is positive 
recurrent, and X1 (11 — A) + Xo2(u2 — TA) < 2. 
In analogy to Example la, we let 


tee min{ u1 — Au, u2 — AU} 
ales Hı + H2 =À 
= mind, p — 5) 


and the corresponding value u* of u is given by 


0 if poy — u2 <-A 
ut = y AG if |u — pa] <À 
1 if pi—=p>A 


For the system with u = u*, X1 + Xo < =. The remark at the end of Example la also carries 
over, yielding that for splitting probability u = u*, X,+ X2 < = 

(b) Consider now the case that when a packet arrives, it is routed to the shorter queue. To be 
definite, in case of a tie, the packet is routed to queue 1. Let Q** denote the transition rate matrix 
in case the route to short queue policy is used. Note that for any u, A(aju + rot) > A(x, A za). 
Thus, (10.16) continues to hold if Q is replaced by Q®°. In particular, if A < py + u2, then the 
process X under the route-to-shorter routing is positive recurrent, and X1 + X2 < os 


2.8. Stability of two queues with transfers 
(a) System is positive recurrent for some u if and only if A1 < 1 +1, A < pa, and A1+A2 < fi + pe. 


(b) 


QV (x) = XO dey (V(y) — V(2)) 
yy Ft 
= Me +1)? 29] + Sle +1? — 23] + Ber — 3 - a) 4 
Pla) —1)2 — 29] + Ee, — 1)? — 28 + (er +1)? 23] (10.17 


(c) If the righthand side of (10.17) is changed by dropping the positive part symbols and dropping 
the factor I;,,>1;, then it is not increased, so that 


QV(a2) < 1101 — u — uv) + z2(à2 + uv — po) + K 
< —(21 + z2) min[ 1 + uv — Az, u2 — Aa — uv + K (10.18) 
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where K = Arda peta To get the best bound on Xj + X2, we select u to maximize the min 
term in (10.18), or u = u*, where u* is the point in [0,1] nearest to patada | For u = u*, we 
find QV (x) < —e(x21 + 22) + K where e = min{ u1 + v — Aq, 12 — Az, ptura], Which of the 
three terms is smallest in the expression for e corresponds to the three cases u* = 1, u* = 0, and 
0 < u* < 1, respectively. It is easy to check that this same e is the largest constant such that the 
stability conditions (with strict inequality relaxed to less than or equal) hold with (A1, Az) replaced 


by (Ai + €, A2 + €). 


2.9. Stability of a system with two queues and modulated server 

By inspection, the equilibrium distribution of the server state is (w(0),w(1),w(2)) = (5, 4, 4)- 
Thus, over the long run, the server can serve queue 1 at most 3 of the time, it can serve queue 
2 at most 3 of the time, and, of course, it can offer at most one service in each slot. Thus, it is 
necessary that aj < 3, a2 < 3, and az + a2 < 1. (It can be shown that these conditions are also 
sufficient for stability by using the Lyapunov function V(x) = ee applied to the time-sampled 
process Y(n) = X(nk), where k is a constant large enough so that the mean fraction of time the 
server is in each state over k time slots is close to the statistical average, no matter what the state 


of the server at the beginning of such interval.) 


3.1. A queue with customers arriving in pairs 


(a) 


A A A 
ola a fk 
NU ATRAEN AN A 


(b) For any function V on Z, QV (x) = A\(Q(a#+2)— Q(z)) + u(Q((a—1)+) —Q(x)). In particular, if 
V(x) = x, then QV (x) = —(u — 2A) + HItz=0}. Therefore, by the Foster-Lypunov stability criteria, 
if y > 2A, then N is recurrent, and if y > 2A, then N is positive recurrent. 

(c) Take V(x) = = Then, 


= x(u — 2A) 4 ‘ate ifxr>1 
iS { 2d ifo=0 
Thus QV (1) < —z(p — 2A) 4 + E for all x € Z. Conclude by the combined Foster-Lyapunov 
criteria and moment bounds that if y > 2A, then N is positive recurrent, and N < OR 


(d) Since the process is not explosive, it is positive recurrent if and only if there is a probability 
distribution p so that pQ = 0. The equation pQ = 0 means that the net probability flux into each 
state should be zero. Equivalently, pQ = 0 means that the next flux out of the set {0,1,...,k} 
is zero for each k > 0. Defining p_; = 0 for convenience, the equation pQ = 0 is equivalent 
to the set of equations: (px-1 + per)A = Pr+1p for k > 0. (This is a set of second order linear 
difference equations.) Multiply each side of the k*” equation by z*+!, sum from k = 0 to oo to 
yield (z7 + zA)P(z) = u(P(z) — po), so that 


= PO 
Fa = 028 


The condition P(1) = 1 yields that po = 1 — 2A/y, which is valid only if 2å/u < 1. Under that 
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condition, 


= 1—2A/p 

1 — (Az/p) — (Az?/p) 
The method of partial fraction expansion can be used to express P as the sum of two terms with 
degree one denominators. First, P is rewritten as 


1—2A/p 
(1 — z/a)(1 — z/b) 
where a and b are the poles of P (i.e. the zeros of the denominator), given by 
4 4 
-1+,/1+ 4 ps =p 
2 o 2 


Matching P and its partial fraction expansion near the poles yields 


Plz) 


P(z) = 


a= 


1 1 
P(z) = (1— 24/n) lz -zA -a O-o- Ti 


so that 


k>0 


AR -k 


(Using this expression we can also find an explicit expression for N, but it is more complicated than 
the bound on N found in part (c).) Thus, we have found that there exists a probability distribution 
p such that pQ = 0 if and only if 2å/u < 1. 

(e) rk = px by the PASTA property. d; is proportional to pz. ip for k > 1, so that dy = 


Pk+1 
1—po ` 


3.2. Token bucket regulation of a Poisson stream 

Suppose for simplicity that the tokens arrive at time 0,1,2,---, and let X; denote the number of 
tokens present at time k— (Another approach would be to consider the number of tokens just after 
time k.) If Bj = E then the one-step transition probability matrix for X = (Xx : k > 0) is 


1— 6o Po 

1 — bo =B1 bı Bo 

1 — bo — = b2 Paz Bi Bo 

P=]. A f 

L= py. =Bp9 Br Bo 

==. efes Bi Bo 
(b) The throughput of packets is equal to the throughput of tokens, where is 1 — 7g. Therefore, 
Pioss for packets = 2058). (The expression used here would be different for the other choice of X 


mentioned in the solution to part (a)). 
(c) Let Z(t) denote the number of tokens present at time t. Then Z is a continuous time Markov 
process with the transition rate diagram shown. 


1 1 1 1 1 
ONO O 
Mw 
A A A A A 
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Then, 


1 AB 


+14 ep BETH HI 


Pross for packets = Plno tokens present] = 


3.3. Extremality of constant interarrival times for G/M/1 queues 

(a) Aj(s) = exp(—s/A). Let tı have distribution function A. Then since exp(—sx) is convex in a, 
Jensens inequality can be applied to yield A*(s) = Elexp(—st1)] > exp(—sE[t]) = A%(s). 

(b) Let co and og be values of ø for the respective queues. Since Az(u(1—0,)) < A* (u(1— 90) = 00 
and A? is convex, it follows (see the figure) that aq < co. The assertions follow because the required 
means are increasing functions of o. 


rh 


A W(1-o) ) 


3.4. Propagation of perturbations 

(a) See the solid lines in the figure. The departure times are indicated by where the solid diagonal 
lines meet the axis, and occur at times 6,8,12,15, and 17. The waiting times in queue are 0, 4, 3, 
0, 1. 


Te T-A Be AS 6 7 8 910: 11-12 “13 “14-15 16° 17 


(b) The departure times are indicated by where the dashed diagonal lines meet the axis, and occur 
at times 6,9,13,15, and 17. The waiting times in queue are 0, 4, 4, 0, 1. 

(c) The departure times are indicated by where the dashed diagonal lines meet the axis, and occur 
at times 6,10, 14, 16, and 18. The waiting times in queue are 0, 4, 5, 1, 2. 

(d) The waiting times of all following customers in the same busy period increase by the same 
amount. Busy periods can merge, and the waiting times of customers originally in a later busy 
period can also increase. 
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3.5. On priority M/GI/1 queues 
(a) The server works on a type 1 customer whenever one is present, and otherwise works on a type 
2 customer, if one is present. If the server is interrupted by a type 1 customer when it is working on 
a type 2 customer, the service of the type 2 customer is suspended until no more type 1 customers 
are in the system. Then service of the type 2 customer is resumed, with its remaining service time 
the same as when the service was interrupted. = 

a a oy XG = ux? 
(b) Let A = A1 + Az, pi = AXi, and p = pı + p2. Then X = Si and X? = Si Type 1 
customers are not effected by type 2 customers, so the mean system time for type 1 customers is 
the same as if there were no type 2 customers: 


MX? — 
2(1 — p1) 
where pı = A¡X1. To find an expression for T2, consider a type 2 customer. Recall that the mean 
work in the system is given by ate ay" The PASTA property holds. Thus, Tə is the mean of the 
sum of the work already in the system when the customer arrives, the service time of the customer 
itself, and the amount of preempting work that arrives while the type 2 customer is in the system: 


T=W +X = 


AX? == 
To = 35 +X + pir. 
2(1 — p) 


Solving for Tə yields 


3.6. Optimality of the uc rule 
Let J(0) = X; AUciWi(o) =>, ucipiWi(co) denote the cost for permutatation ø. 

Claim: Let o be a permutation and let G6 denote another permutation obtained from o by 
swapping o; and 0;,1 for some i. Then J(0) < J(c) if and only if Ho;Co; < Mor Coins 

The claim is proved as follows. Since only customers of classes i and ¿+ 1 are affected by 
the swap, W;(¢) = W;(0) for j ¢ {o;, 0:41}. On the other hand, since class o; (i.e. class 041) 
customers have lower priority under ô than under o, Wa; (ô) > W,, (0). Thus, 9 > 0, where 


ô = Po; Wo; (ô) = Po; Wo; (0). 
Similarly, Wo,,,(¢) < Wo,,, (0), and furthermore by the conservation of flow equations, 
Por Wo;z1 (5) — Poia Weiss (0) = —d. 


Therefore, J(9) — J(0) = (o,Co, — Moi+1Co,,1)0- Since 6 > 0, the claim follows immediately. 

By the claim, if ø does not satisfy the ordering condition y,Co, > Moros > +++ > Mog Cox, then 
o is not optimal. 

Conversely, if o does satisfy the ordering condition and if o? is an arbitrary ordering, there is 
a sequence of orderings 0°,c1,...,0? =o so that J(0%) > J(o!) > --- > J(0P) = J(0), so that o 
is optimal. 

The pc rule is intuitively plausible because uc is the rate that instantaneous cost is reduced 
when a rate u server works on a customer with instantaneous holding cost c. 
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3.7. A queue with persistent customers 

(a) The system is stable if and only if 22 < 1. See part (b) for justification. 

(b) Under the PR service order, we can view the network as an ordinary M/GI/1 queue in which 
the service time variable X for a typical customer is D times a Geo(b) random variable. The system 
is stable if p < 1, were p = AX = àD, Under this condition, the mean time in the system is given 


by the Polleczek-Khinchine formula with p = Ap X= p, and X? = pa, 


= AX? > (2—dD)D 
eT Say U.S AD) 


(c) The distribution of the number of customers in the system does not depend on the order of 
service (assuming no advance knowledge of how many times a given customer will return). So the 
mean number of customers in the system does not depend on the service order. So by Little’s 
law, the mean time in the system does not depend on the order of service. However, the FIFO 
order at the queue would lead to a larger variance of total system time than the PR service or- 
der. (This is implied by the fact that if aj < --- < an and di < +++ < dn and a; < dj, then 
YN (d; — ai)? < Y ds, — ai)? for any permutation o other than the identity permutation. Note that 
the PR order is not FIFO for the queue, but is FIFO for the system.) 


3.8. A discrete time M/GI/1 queue 

The mean service time is 7/2, so the service rate is 2/7. Let p = 7p/2, which is the arrival rate, p, 
divided by the service rate. Also, p is the probability the server is busy in a time slot. Arrivals see 
the system in equilibrium by the discrete time version of the PASTA property, which is true when 
numbers of arrivals in distinct slots are independent. Given the server is busy during the slot of 
an arrival, the residual service time y of the customer in service is distributed over {0,1,2,3,4,5}. 
The total service time of the customer in service has distribution P[l = l] = L, for 1 < l < 6, 


5 2 2 2 2 2 2 
with mean E[L] = ™ = 1?+2 J ea 


Ely] = aa z 5, Thus, W = 3.5N + (DE), where N is the mean number of customers seen 


by an arrival, equal to pW. Solving for W yields W = 5 forO<p< 2. An embedded Markov 
process is given by the number of customers in the system just after the departure of a customer. 
To be specific, this number could include the new arrival, if any, occurring is the same slot that a 


service is completed. 


. Given L, y is uniform on {0,1,..., L — 1}, so 


Here ia a second way to solve the problem, obtained by viewing the events at slot boundaries 
in a different order. Suppose that the customer in service exits if its service is complete, and if so, 


the customer at the head of the queue (if any) is moved into the server before a possible arrival is 
added. In this view, a new customer still arrives to find the server busy with probability p = w. but 
the residual service time is distributed over {1, 2, 3,4,5,6}. The total service time of the customer 
in service hs the same sampled lifetime distribution as before , with mean E[L] = B, but here, 


given L, y is uniformly distributed over {1,..., L}, so Ely] = 5, Thus, by the PASTA property, 


W = 3.5N + DE, where N is the mean number in the queue seen by an arrival. A customer 
waiting W slots for service can be seen in the queue for W — 1 observation times, so by Little’s law, 


N =(W-1)p. thus, W = 3.5(W —1)p+(#)(8), and solving for W yields W = 5 for0<p< 2. 


3.9. On stochastic ordering of sampled lifetimes 
If fL < gz, then ie (x) fr(a)dx > dae d(x)9,(1)dx for any strictly decreasing positive function $ 
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on Ry. Taking $(x) = 4 yields that 1 = f>° f(x)dx > fy g(a)dx = 1, which is impossible. 
3.10. Effective bandwidth, bufferless link 

(a) a(s) = ($), a(s) = HE, 

(b) n=79.05 (or round to n = 79) 

(c) a1(s*) = 1.05, ag(s*) = 1.19, and Ceff=177.0. As one might expect, 01 (s*) is somewhat smaller 
than a2(s*). 


3.11. Effective bandwidth for a buffered link and long range dependent Gaussian 
traffic 
(a) n = 88.80 (round down to n=88). a1(s*) = 1.0085, ag(s*) = 1.1354, t* = 31.6, Ceff = 190.38. 


(b) n = 85.33 (round down to n=85). a1(s*) = 1.015, ag(s*) = 1.1827, t* = 22.7, Ceff = 187.53. 
Due to the tighter overflow probability constraint, the effective bandwidths in part (b) are larger 
than those in part (a) and the effective bandwidth of the link is smaller. Interestingly, the critical 
time scale is smaller for part (b) than for part (a). 


3.12. Extremal equivalent bandwidth for fixed mean and range 

Intuitively, X should be as bursty as possible, suggesting the choice P[X* = 0] = P[X* = 2] = 0.5. 
Here's a proof. Fix s > 0. The function eĉ” is convex in x over the interval x € [0, 2], with values 1 
and e?* at the endpoints. So e** < 1+ $(e?* — 1) for x € [0,2]. Plugging in X on each side, taking 
expectations, and using the constraint E[X] = 1 yields Ele*%] < kii =p lee |) 


5.1. Time reversal of a simple continuous time Markov process 


-1 


3.0. 0 0 ¿£ 0 0 0 
1 1 

Q = 0 3(l+a) o 0 QT 0 3(1+a) 0 0 
0o 0 4 0 0 0 4 0 
0 0-0 a O° Ope Oy aes 
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5.2. Reversibility of a three state Markov process 

The process is reversible and m = (7,,7r3,73) is the equilibrium distribution, if and only if the 
detailed balance equations hold: mı = ara, 72 = 613, 73 = cri. The first two of the equations yield 
that mı = abr3, which can hold together with the third equation only if abc = 1. This condition 
is thus necessary. If abc = 1, we find that the detailed balance equations hold for the probability 
distribution proportional to (ab, b,1). Thus, the condition abc = 1 is also sufficient for reversibility. 


5.3. A link with heterogeneous call requirements 
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(b) If there were infinitely many channels, the numbers of slow and fast connections would form 
independent M/M/oo systems, which is reversible, because each coordinate process is a one- 
dimensional, and hence reversible, Markov process, and two independent reversible processes are 
jointly reversible. The equilibrium distribution would be z(i,j) = A The process 
with the finite number of channels is obtained by truncation of the process with infinitely many 
channels, to the state space S*” = {(i,j) € Z? :i+ Lj < KL}, and is hence also reversible. The 


AS EN where Z(K, L) is selected to make r%* sum to 


equilibrium distribution is 144 (i, j) = DARDO 


one 


(c) Bs = DL ymL(K — j)L, j), and Bp=1- 4D. 


Z(KL) 

5.4. Time reversibility of an M/GI/1 processor sharing queue 

(a) No, due to the memory in the service time varibles. One Markov process is T(t) = (N(t), Si(t),... 
where 51(t) > --- > Sy)(t)) > 0 and S;(t) denotes how much service the ith customer in the sys- 
tem received so far. 


(b) Let n = (n7,...,nx) and n = nı +---+n,. The nonzero off-diagonal rates are 
Tiki ifn’ =(n1,...,ni— 1,841 +1,..., Nk) 
q(n,n!) = à ifn’ = (m +1,n2,..., Nk) 
Tkhk ifn! = (ni, see yg Nk—=1; nk — 1) 
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(c) The conjectured rate matrix for the time-reversed process is given by (only nonzero off-diagonal 
rates are indicated): 


a Hi if n! = (nı, sag jj as ng) 
Ilan! . 
q (n ,N) = Can ifn’ = (nı +1,n9,... , Nk) 
A if n! =(n3,...,ny-1,N4 — 1) 


To simultaneously prove the conjecture for 7 and q”, it suffices to check 1(n)q(n,n’) = r(n')q (n'n) 
for n £ n'. We find 


mn’) _ mate = rahi i i a (m1,. ae = 1, niyi E 1,..., Nk) 
m(n) mae = para +) i E = (ni +.1,Nne,..., 7k) 
pnay An if n’ = (Ni sg MER HE 1) 
_ nn) 
5 q (nn) 


as required. 

(d) The total service time for a customer has the same distribution in either system, namely the 
sum Exp(u1)+-::-+ Exp(px). Hence, the random processes giving the number of customers in the 
system are statistically the same for the forward and reverse time systems. 


5.5. A sequence of symmetric star shaped loss networks 

(a) The set R of routes is the set of subsets of {1,..., M} of size two. Thus, |R| = (Y: The 
process at time t has the form X(t) = (X, (t) : r € R) with state space S = {x € ZR: pier Ur E 
5 for 1 <i <M} and transition rate matrix given by (only nonzero off diagonal rates are given): 


qlx,t+er)=>x5 Erster ES 
q(£, £ — er) = Tr lfx,r—e, €S 


(b) The unthinned rate at a link £ is given by Ae = Eppen ve = Hr : r € LH x gro = 4, so for 
any M, Be < El4,5] = 0.19907 and the call acceptance probability is greater than or equal to 
(1 — 0.19907)? = 0.64149. 

If M = 2 then By = El4,5] and the call acceptance probability is 1-0.19907=0.8001. 

(c) The reduced load approximation is given by the fixed point equations: 


Ne= (1 — Bz) and By = El\p,C] 


The equations do not depend on M, and the solution is Bo = 0.1464, leading to the estimate 
(1 — By)? = 0.72857 for the call acceptance probability. The fixed point approximation for this 
example, for both B: and call acceptance probability, can be shown to be exact in the limit M — oo. 
The numbers are summarized in the following table: 


B; P{route r is free} 
bound < 0.1999 > 0.6915 
M=2 0.199 0.811 
M=3 0.14488 0.745 
M=4 0.146 0.739 
M — œ 0.146 0.728 
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6.1. Deterministic delay constraints for two servers in series 

(a) The total arrival stream to the first queue is (01 +02, p1 + p2) constrained, and the server is 
FIFO, so the delay is bounded above by dı = pza], (dı = 2 for the example.) 

(b) Stream 1 at the output of queue 1 is (01 + p1di, p1)-upper constrained, because A is (01, p1)- 
upper constrained and delay in the first queue is less than or equal to dı. (o = 01 + pid; = 8 for 
the example.) 

(c) By the example after Propositoin 6.3.2, there is a bound on the delay of the lower priority 
stream for a FIFO server, constant service rate C, and (0, p) constrained inputs. This yields: 


(o1t+pidi)+o3 12 


=4 (for the example 
e 3 ( ple) 


delay for stream 1 in queue 2 < 


6.2. Calculation of an acceptance region based on the SCED algorithm 

(a) Let g(t) = o + pt for p,o > 0. By the definition given in the notes, g*(0) = 0 and for t > 1, 
g*(t) = min{g(s1) +: +g([spn):n> 1, $1,...,8 21, s1 +--+ Sn = t} = minfon + pt: n > 
1} = ø + pt. Another approach to this problem is to use the definition of g* in [5]. Yet another 
approach is to let go(t) = L(4>1,9(t) and show that go is the maximal subadditive function equal to 
0 at 0, and less than or equal to g. 

(b) Before working out special cases, let's work out the general case. Let g(t) = o + pt and 
f(t) = (t—d), where o > 0,D >0,r > p> 0. To find g* x f by a graphical method, we use the 
formula (g* x f)(t) = mino<s<t{g*(s) + f(t — s)}. That is, g* x f is the minimum of the functions 
obtained by considering g*(s) + f(t — s) as a function of t on the interval t > s for each s fixed. 


So, for the parameters at hand, 


(c) Since nig} x fi + nag3 * f2 is piecewise linear with intial value 0, this function is less than or 
equal to Ct for all t > 0 if inequality is true at breakpoints and at +00 (i.e. in the limit as t — 00). 
This requires n1(25, 36, 40, 1) + na(0, 0, 24,4) < (900, 2000, 2400, 100), which simplifies to nı < 36 
and nı + 4n2 < 100. 
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6.3. Serve longer service priority with deterministically constrained arrival processes 
(a) The two queues combined is equivalent to a single queue system, with a work conserving 
server that serves up to C customers per slot. The total arrival stream is (0, +02, p1 + p2)- upper 
constrained, so that the maximum combined carry over for a given slot t to the next satisfies 
q(t) < 01 +02. 

(b) By the same reason, the delay bound based on system busy periods can be applied to yield 
d< | tos | 

cae ae pi iS gaan 

(c) Under FIFO service within each stream and service to longer queue, each customer from stream 
1 exits no later than it would exit from a system with the same arrival streams, FIFO service within 
each stream, and pure priority to type 2 customers, a case considered in the notes and class. The 
delay is thus bounded above by Oop! 


7.1. A shortest path problem 
(a) 


28) 


E 3) 


6 


(b) Yes, because D; > D; + di; for every edge ij not in the tree. 
(c) Synchronous Bellman-Ford would take 8 iterations (on the 9th iteration no improvement would 
be found) because the most hops in a minimum cost path is 8. 


7.2. A minumum weight spanning tree problem 


The MWST is unique because during execution of the Prim-Dijkstra algorithm there were no ties. 
(The sufficiency of this condition can be proved by a slight modification of the induction proof used 
to prove the correctness of the Prim-Dijkstra algorithm. Suppose the Prim-Dijkstra algorithm is 
executed. Prove by induction the following claim: The set of k edges found after k steps of the 
algorithm is a subset of all MWST’s.) 


7.3. A maximum flow problem 
The capacities, a maximum flow, and a minimum cut, are show. The maximum flow value is 15. 
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8.1. A flow deviation problem (Frank-Wolfe method) 

(a) Initially the 8 links directed counter clockwise along the perimeter of the network carry zero 
flow, and the other 16 links carry flow b. So cost(f)=8b?. 

(b) Flow f uses the 8 links not used initially, as shown. 


(a) (b) 
4 y Y 
A 
4 1] 
y 


y 


Thus, cost(af + (1 — a)f) = 8(2ab)? + 16((1 — a)b)?. Minimizing over a, we find a* = E. Under 
the new flow, fı = a* f + (1 — a*) f, all 24 links carry flow 2, The first derivative link length of all 
paths used is minimum. So fı is an optimal flow. 


8.2. A simple routing problem in a queueing network 


In equilibrium, the stations are independent and have the distribution of isolated M/M/1 queues, 
1 


and the delay in an M/M/1 queue with arrival rate A and departure rate p is EN Therefore, 


1 


1 1 1 
Do a) (tp) 


(b) 


p=0 p=0.17 


ole oe ee 


(c) p = 0 minimizes Da, because Da is increasing in p. 
(d) Setting Dj(p) = 0 yields p = 3. 
(e) Setting D} (p) + D; (p) = 0 yields p ~ 0.17. Only one iteration of the flow deviation algorithm 
is necessary because the only degree of freedom is the split for flow b between two paths. The line 
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search for flow deviation solves the problem in one iteration. 
(£) The solution is p = $, because Da is increasing in p, Dj is decreasing in p, and Da(3) = Dy(3)- 
(It is a coincidence that the answers to (d) and (f) are the same.) 


8.3. A joint routing and congestion control problem 
(a) Since Di(F’) = F, the price for a link is equal to the flow on the link. For example, path 124 
has price 2a + d. The optimality conditions are: 


2a+d > Bra with equality if a > 0 

a+b 
Pia, rae 

2b+c > with equality if b > 0 
a+b 

2c+b > Ba with equality if c > 0 
c+d 

a +2d Bas with equality if d > 0 
c+d 


The first two conditions are for o-d pair w = (1,2) and the last two are for o-d pari w = (2,3). 
(b) If a,b, c,d > 0 the equality constraints hold in all four conditions. The first two conditions yield 
2a+d = 2b+c or, equivalently, 2a — 2b = c—d. The last two yield 2c+ b = a+ 2d or, equivalently, 
2a — 2b = 4c — 4d. Thus, c— d = 4c — 4d or, equivalently, c = d, and hence also a = b. (c) We seek 
a solution such that all four flows are nonzero, so that part (b) applies. Replacing b by a and d by 
c, and using 914 = 66 and 623 = 130, the optimality conditions become 


66 
2 = — 
a+c a 
130 
2c+a = — 
2c 


or a=b=3 and c=d=5. To double check, note that the two routes for w = (1,4) have price 11 
each, which is also the marginal value for route (1,4), and the two routes for w = (2,3) have price 
13 each, which is also the marginal value for route (2,3). 


8.4. Joint routing and congestion control with hard link constraints 
(a) The primal variables should be feasible: a,b,c,d > 0 and a < C1,b +c < C2,d < C3. The dual 
variables should satisfy the positivity and complementary slackness condition: p; > 0 with equality 
if link 7 is not saturated. It is easy to see that the optimal flow will saturate all three links, so that 
nonzero prices are permitted. Finally, since Ui (11) = x! 2 and Us(x2) = In(x2), the remaining 
conditions are: 

(a+b)12< pı with equality if a > 0 

(a+b)12< pọ with equality if b > 0 

sq < P2 with equality if c > 0 

a <p3_ with equality if d > 0 


c+ 


(b) Clearly we can set a = d = 8 and c= 8 — b. It remains to find b. The optimality conditions 


become 
(8+b) 19 < po with equality if b > 0 


== < po with equality if b < 8 
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Since (8 + b) 1/2 > ai; over the entire range 0 < b < 8, the optimal choice of b is b = 8. This 
yields the assignment (a,b,c,d) = (8,8,0,8), price vector (p1, p2, p3) = (4, 17, and maximum 
value 2 + In(8). 


8.5. Sufficiency of the optimality condition for hard link constraints 
Since L(x,p) is concave in x, for any e with 0 < €< 1, 


L(x,p) — L(0*,p) <= (le + ew — 2°),p) — La", p)) 


Taking the limit as e — 0 yields 


L(<,p) — L(a*,p) < SOU; (ap) (ae — T Darl Y 2 — a 


r:ler 


Le- 2) la - E») <0 (10.19) 


r ler 


where the final inequality follows from the fact that for each r, the quantity in braces in (10.19) is 
less than or equal to zero, with equality if xš > 0. 
(b) Note that X; pi(Ci — ler 27) = 0 and if Ax < C, then $7, 91(Ci — X rrer Tr) 2 0. Soifx>0 


and Ar < C, 
S_U (at) = L(x*,p) > L(a,p) > ed 


8.6. Fair flow allocation with hard constrained links 
(a) By inspection, tmaz—min = (3, 3, z, 2). 
(b) (proportional fairness) Let p; denote the price for link J. Seek a solution to the equations 


1 1 1 1 


= “<= T2 = T3 TA = 
pı + p2 + p3 Pı + p2 P1 P2 + P3 
11 + £2 + z3 < 1, with eqaulity if pı > 0 


11 + £2 + x4 <1, with eqaulity if po > 0 
11 + x4 < 1, with eqaulity if p3 > 0 


cee 11 +24 < 1, so that p = 0. Also, links 1 and 2 will be full, so that 13 = x4. But £3 = > and 


z4 = =, so that pı = pz. Finally, use opr + opr + a to get pı = p2 = 2, yielding tp, = (4, 7 3, 3). 


Figs 1 and 2 use paths with price pı + po = 4 and each have rate L. 
Flows 3 and 4 use paths with price pı = p2 = 2 and each have rate > 


9.1. Illustration of dynamic programming — a stopping time problem 

(a) Consider the game for a possible n + 1 observations. After seeing X1, the player can either sop 
and receive reward X ly or continue and receive expected reward Vp. Thus, Vz41 = E[X1 V Vn] = 
Jo @v Vaje e*de = fir Vne "da + fy xe~*da = Va qe Y, 

(b) The optimal policy is threshold type. For 1 < k < n— 1, the player should stop after observing 
Xx if Xk > Vox. The rule for n = 8 is pictured. 
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7 V6 X4 
eo. 
2 eae 
x e 
e “el Vy 
x X e 
1 e! 23 A 


A A eh hee 
1 | 1.00000 1.00000 
2 | 1.36788 1.50000 
3 | 1.62253 1.83333 
4 | 1.81993] 2.08333 
5 |1.98196| 2.28333 
(c) 6 | 2.11976} 2.45000 
7 | 2.23982} 2.59286 
8 | 2.34630} 2.71786 
9 ) 2.44202] 2.82897 
10 | 2.52901 2.92897 
20 | 3.12883 | 3.59774 
30 | 3.49753 |  3.99499 


9.2. Comparison of four ways to share two servers 

(a) System 1: Each of the two subsystems is an M/M/1 queue, so that N; is twice the mean number 
in an M/M/1 queue with arrival rate A and departure rate u: Ni = ey 

System 2: Each of the two subsystems is G/M/1 where the interarrival distribution is the same as 
the sum of two exponential random variables with parameter 24. We will analyze such a G/M/1 


queue. The Laplace transform of the interarrival distribution is A*(s) = (JS ete gos = 
2A 
Ga 
0 — 2(1 + 2p)o? + (1 + 4p(1 + p))o — 4p? = 0. Since o = 1 is a solution, this equation can be 
written as (o —1)(0?— (14 4p)0 +4p?) = 0 which has solutions o = 1 and o = ae Ep VETS? Thus, 
1+4p—./1+8p 
oe 


)?. We seek the solution ø in the range 0 < o < 1 of the equation o = A*(u — uo), or 


there is a solution ø € [0,1) if and only if 0 < p < 1, and it is given by o = Therefore, 


ae] A 4p 
N= eg 1—4p+,/1+8p" 
System 3 The third system is an M/M/2 queueing system. The number in the system is a birth- 
death Markov process with arrival rates Az = 2A for all k > 0 and death rates uz = (k A 2)u for 


k> 1. Let p= A The usual solution method for birth-death processes yields that pı = AA po and 


Pk = 2p"po for k > 1. The process is positive recurrent if and only if p < 1, and for such p we find 
po = (1 +2p++-))= i and N3 = Tae which is smaller than N; by a factor 1 + p. 

(b) For 0 < p< 1, N3 < N¿< No < Ni. Here is the justification. First, we argue that N3 < Na, 
using the idea of stochastic domination. Indeed, let there be three independent Poisson streams: an 
arrival stream A with rate 2, and for i = 1,2 a potential departure stream D; with departure rate 
li. Consider systems 3 and 4 running, using these same three processes. If stream D; has a jump 
at time t, then there is a departure from system 3 if queue 2 is not empty, and there is a departure 
from system 4 if the entire system is not empty. By induction on the number of events (arrival or 
potential departure events) we see that the number of customers in system 3 is always less than 
or equal to the number of customers in system 4, and is occasionally strictly less. Thus, N3 < Na. 
We argue using dynamic programming that system 4, using send to the shorter queue, is using the 
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optimal dynamic routing policy. Thus, N4 is less than Ni and No. Finally, we can compare the 
formulas found above to see that Nı > N2. ( It can be shown that as p > 1, (1 — p)M => 3> 


(1 — pNa => 2, (1— p)N3 > 1, and (1 — p)N4 > 1. ) 


9.3. A stochastic dynamic congestion control problem 

We take the state space to be S = {0,1,..., K}, where a state represents the number of customers 
in the system. Let a, with a > 0, denote the discount rate. The interevent times are exponentially 
distributed with parameter y = À + ào + u, and the control set is given by U = [0,1]. The one step 
transition probabilities are given by (the quantities in the diagram should be divided by y): 


ħothu Àothu Aotà u Aothu Aotu Aotà u 
¡O OMEN a a > PAT > 
0) 1) (2) ( N-2 NI) (n) 
SN A E ASF m > PEPA 

t T) file ae uf | a E 


NU NJ y UW VW J 
u+à(l-u) A(l-a) A (l-u) (l-u) Aà(l-u) Roth 


p+A(l-=u) A+ Au 
u A — u) A+ Au 
u Al — u) o+ Au 


The cost is given by 


= —at 
cost = EN a and an arrival occurs at time y ~ {Ae:=-1} | € ú 
t>0 


CO 
Ef e ox; uz)dt ae 
0 a 


where g(z,u) = c(ào + Au) ¡»xy + 7L(2=0)- Here we decided that u can be nonzero even when 
the system is full. It is optimal to take u, (K) = 0 however, because admitting customers when 
the system is full incurs a cost with no change in state. Therefore, we could have assumed that 
un(K) = 0 and taken g(x, u) = Aol {2z=K} +7 HL fo=0) We will drop the constant term, and instead 
take the cost to be 


OO 
cost = Ef e g(az, uz)dt 
0 


(b) For the discrete-time equivalent model, the one-step discount factor is 3 = ae and the value 


+ 
update equations are given by 


Vaqi(2) = oe la, u) + f {uVna( (x£ — Dy) + ào Vn ((£ +1) A K) +A(1— wu) Vp (x) + AuV, ((£ +1) A K)} 


In particular, for x = K, 


Vaio = min cir. + Au) + - pr, (K-D + (Ao +00) 


O<sus 
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so we see that indeed u = 0 is optimal when x = K. Thus, using g(x, u) = CAolta=ky + 7 HL to=0y- 
we have that 


Va +10) = g(a) + : [UVn((x = 1)+) + AoVn((@ +1) A K) + Amin (Vp(2), Va((x+1) AK)y] 


(c) When there are n steps-to-go and the current state is x, an optimal control is given by 


O vaa SV et AE) 
un[r) = { i VG) Ss Vler DKK) 


(d) Let us prove that for each n > 1 there exists a threshold 7, such that un(1) = Lio<r,y- It is 
enough to show that the following is true for each n > 1: 

(a) Vn is convex, i.e. Va (1) — Vn (0) < Vn(2) — Va(1) < ++: < Va(K) — Vna(K — 1) 

(b) —"F < Va(1) — Va(0) 

(c) Vo (1H) = Vi (KE = 1) < F 
To complete the proof, we prove (a)-(c) by induction on n. The details are very similar to those 
for the Meontrotlea/M/1 example in the notes, and are omitted. 
(e) For an exact formulation, we need to describe a state space. If FCFS service order is assumed, it 
is not enough to know the number of customers of each type in the system, so that the state space 
would be the set of sequences from the alphabet {1,2} with length between 0 and K. We expect 
the optimal control to have some monotonicity properties, but it would be hard to describe the 
control because the state space is so complex. If instead, the service order were pure preemptive 
resume priority to customers of one type, or processor sharing, then the state space could be taken 
to be {(n1, n2) : nı > 0,n2 > 0, and nı + nz < K}. We expect the optimal control to have a 
switching curve structure. 


9.4. Control dependent cost 


(a) 


ba n-1 tht 
E, T Aes = SE, dl AX, UD) Td 
o k=0 tk 
n-1 thai 
SS —at 
2 Lef g(X (ty), wpe di 
n-1 that 
= Y Exlo(X (ta), o) f e dt] 
k=0 te 
n-1 that 
= Yall X(t), wE edt 
A n—1 
= = BY g(X (tk), We) 
k=0 


where we used the fact that 


ef ede] = EN (e72 = e tk+1)) = Lø _ are = a- ae 


> 
Q 
Q 
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(b) The only change is that g should be added to the cost before the minimization over u, yielding: 
Vasala) = inf [ateo +A rat va( 


9.5. Optimal control of a server 
(a) We use S = Z2, where state x = (11,12) denotes x; customers in station i, for i = 1,2. Let 
Aı(z1, 22) = (11 + 1, 22) and D¡ (11,12) = ((11 — 1)4, z2), and define Ag and Də similarly. Then 


2 
_ 1 
Pay (u) = = >> (Alty) + mil) Ly=D,(0))) 
i=l 


ee 


The CDF F corresponds to the exponential distribution with parameter y = A1 + A2 + uo + u1 + be, 
and the instantaneous cost function is g(x) = 11 + 22. 
(b) The backwards equation of dynamic programming becomes 


2 
Vasile) = gle) + min, PY Va (Aila)) + mat) Va (De) 


Susi y E 
or, after plugging in the optimal value for u, 
2 
Vn4i(x) = aly (AiVn( ) + pu(u) Va (Di (2))) + o min[V, (D1(2)), Va(D2(x))} 
i=1 


with the intial condition Vo = 0. 
(c) 


u (10.20) 


* (q) = 1 if V, (Dix) < Va( Dax) 
n Y 0 else. 


(d) We conjecture that the service rate uo should be allocated to the station with the longer queue. 
That is, uz (£) = Lx, >0,). Equivalently, V (x1, 12) < V(x1 — 1,12 + 1) whenever 0 < x] < x2 and 
V(a1,22) < V(x, + 1,22 — 1) whenever 0 < 22 < 21. 


9.6. A dynamic server rate control problem with switching costs 

(a) We use the state space S = [(1,0) : 1 € Z4,0 € {H,L}}, where for a given state x = (1,0), 
l denotes the number of customers in the system and @ denotes the state of the server. Let 
y = A+ up, which is the maximum event rate that we shall use. (Another natural choice would 
be y = à + uL + uy, and then the self loop transition probabilities would be increased in the one 
step transition probabilities.) We take the control values u = (ua, ug) to be in the set U = [0, 1]?, 
where ua is the probability the server is in the high state after the next event, given the next event 
is an arrival, and ug is the probability the server is in the high state after the next event, given the 
next event is a departure. The following diagram gives the transition probabilities (the quantities 
shown should be divided by y).? 


Variations of these equations are possible, depending on the exact assumptions. We chose to allow a server state 
change at potential departure times, even if the potential departures are not used. 
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The total costs can be described as follows. 


T 
cost due to use of high service rate and queueing = Ey f {cu Io, =H} + arta 
0 


cost due to switching = Ey, S estío, Lo =nye 
t<T 


T 
Ex f csIto, =L} [Aua(t) + palio ta 
0 


Thus, the cost is captured by taking 
g(x,u) = g((1,0), (ua, Ua)) = cul gan} + cwl + CsL(9=1y [Ma + urua] 


(b) We first consider V,, +1 (1) for states x = (1, 0) with 6 = H. Since g((l, H), u) = cy + cwl, which 
doesn't depend on u, the backwards recursion is: 


if A(1 — ua) 


Vi4ill, H) = cu + cwl + 68 min {eva 1,40 
u€[0,1]2 Y 


H 9, m+ HH td y, 1)4,2)] 


The analogous equation for 0 = L is 


y Aua 
Vn+1(l, L) = lO? cwl + cs{Aua F HLUa} + 4 > 


ra 0 + BPI 0) 


Solving for u in the above equations and setting 6 = => yields the following simpler version of the 
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dynamic programming backwards recursion: 


o =) 2a ape = min{V,(I + 1, H),Va(I-+1,L)} (10.21) 
+ SEH in (Va 1) 44H), Va Ds D) 

A Seg: = ORUCA AN (10.22) 
+ ME mine eV) Di Da Puu ely q, D 


The optimal controls can be succinctly described as follows. If the current switch state is 0, and 
if there will be 1! customers in the system after the next arrival or potential departure, then the 
optimal server state after such arrival or potential departure is given by: 


If 9 = H then: Va (l, L) > Val, H) < new server state H 


If 0 = L then: Vall, L) > Va(U,H) +8 <= new server state H (10.23) 


The function V,,(1, 0) represents the cost-to-go for initial state (1,0), with the understanding 
that it is not possible to switch the server state until the first event time. Let W,,(1, 0) denote the 
cost-to-go, given the initial state is (1,0), but assuming that the server state can be changed at 


time zero. Then the dynamic programming equations become:? 
W,(1,0) = minfV,(!, L), Va (1, H) + OL 913) (10.24) 
A 
Va+1(1,0) = culjo=Hy + cwl+ mal +1,0) + Pe w -= 1) 58) 
+ ee {Irou} Wn(l — 1, H) + Lg9=1Wn(1, L)} (10.25) 


with the initial conditions Vọ = 0 and Wn = 0. 
In the special case that there is no switching cost, 6 = 0. In that case, Wn (l, 0) does not depend 
on 0, so we write it as W,,(1). Then the dynamic programming equations become: 


Wh = BED. va) (10.26) 
v E PA Pur 
n+1(0,0) = culo=Hy + ewl + al +1) + a =1)4) 


T Heoi {Ion} Wn(l —1)+ Io=1}Wn (0) } (10.27) 

In the case of no switching costs, it is rather simple to show by induction on n that Wn is a convex, 

nondecreasing function on Zy and Vn satisfies properties 1-5 in the original solutions. As noted in 
the original solutions, property 3 for V, implies the conjectured threshold structure. 

(c) We conjecture that the optimal control has a threshold behavior, because when there are 

more customers in the system, it is more valuable to quickly serve the customer in the server, 

thereby reducing the waiting times of all remaining customers. Since there is a switching cost, 


3In this version, we allow a change of server state after any event. Thus, the functions V, are slightly different 
than the ones above. 
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however, it is better to be a bit reluctant to switch, because excessive switching is costly. This 
suggests that the threshold for switching from H to L should be smaller than the threshold for 
switching from L to H. Specifically, we expect there are two thresholds, Ty and Tz, with TH < TL, 
so that the optimal control law (10.23) becomes 


l >To <= new server state H (10.28) 


This leads to the nonzero transition probabilities shown. The transient states are included in the 
first diagram and are omitted in the second. 


DN os ‘a xy” ane Y aie eS E BO NE ADE ae 
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T T 
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We haven’t found a proof of the conjectured threshold structure in general, but we here sketch a 
proof in the case of zero switching cost (6 = 0). Consider the following properties of a function V 
on Zy x {L, H}: 


1. V(1,0) is nondecreasing in l for 9 = L and for 0 = 

2. V(l,0) is convex in l for 0 = L and for 0 = H 

3. Vil+1,L) -V(U+1,H) —V(l,L)+V(l, H) > 0 for 1 > 0. 

4. V(1+2, H) —V(l+1,H)-—V(l+1,L)+V(1,L) >0 forl>0. 
5. V(1+2,L) -V(I+1,L) -V(U+1,H) + V (1, H) >0 for 1 >0. 


We write property 1.H to denote property 1 for 6 = 1. Properties 1.L, 2.H, and 2.L are defined 
analogously. The following figure gives a graphical representation of the five properties. 


Property: 1L 1.H 2.L 2.H 3 4 5 
H -+ + 2-+ + - -+ + - 
E -+ + 2-+ -+ + - -+ 


For each property, the figure indicates the type of linear combinations of values of V which should 
be nonnegative. The collection of properties 1-5 are nearly symmetric in H and L. Property 5 is 
obtained by swapping L and H in property 4. Thus, property 3 is the only part of properties 1-5 
that is not symmetric in L and H. 

Properties 3 and 4 imply property 2 (both 2.L and 2.H). This can be seen graphically by sliding 
the diagram for property 4 to the left to overlap the diagram for property 3, and adding. Similarly, 
properties 2.L and 3 imply property 5. Thus, properties 3 and 4 together imply properties 2 and 
5. So to prove a function has properties 1-5, it suffices to prove it has properties 1, 3, and 4. 

Property 3 is the one connected to the threshold structure. Another way to state property 3 
is that V (l, H) — V (L.L) is nonincreasing in l. That is, as l increases, given there are l customers 
in the system, it becomes increasingly preferable to be in state H. More specifically, if Vp satisfies 
property 3, and if 7, = max{l : V(L, H)—V(l, L) > 0} and ty = max{l : V (1, H)+8-—V(L, L) > 0}, 
then (10.23) is equivalent to (10.28). 

It remains to show that V, has properties 1-5 for all n > 0. To do so, it can be proved by 
induction on n that V, has properties 1-5 for all n > 0, and W, is convex and nondecreasing 
on Z4. For the base case, observe that the function Vo given by Vo = 0 has properties 1-5. 


195 


For the general induction step, it can be easily shown that if V,, has properties 1-5, then W,, is 
convex, nondecreasing. And given that Wn is convex, nondecreasing, it can be shown that Vn+1 
has properties 1-5. As mentioned above, it suffices to establish that V, +1 has properties 1,3, and 
4. Unfortunately, this approach doesn't seem to work in case 6 > 0. 

(d) First, if cg = 0 then the cost-to-go does not depend on the server state, and in particular 
the thresholds should be equal: 7, = TH. Suppose a single customer is served, with no other 
customers waiting or arriving. If the server is put into the high state, and if ty denotes the time the 


customer departs, then the average cost is El ¿"e ““dt](cy+cw) = ERA (cy +ew) = O 


Similarly, if the server is put into the low state during the service interval. then the cost is > ie 
The given inequality implies that it costs less to use the high rate, even if there is only one customer 
in the system over all time. But using the high rate for a given server helps decrease the costs for 
any other customers that might be in the system, so that it is optimal to always use the high rate 


(when the system is not emput), under the conditions given in the problem. 
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